Tags:
create new tag
view all tags

Arrow left Go to previous page / next page of CMS site log MOVED TO...

25. 09. 2009 Retransfer of missing blocks of Summer09 datasets

List of lost blocks: lostBlocks_summer09.list: list of lost Summer09 datasets blocks

  • h 11:42: Requested first part of lost Summer09 lost blocks, size ~1TB, for testing purposes. Request 62359:

https://cmsweb.cern.ch/phedex/prod/Request::View?request=62359&.submit=Submit

  • h 12:00: Prod Agents stopped in order to manually clear lost pnsfs, as they are hanging jobs on dCache
  • h 14:00 - 26/09/2009 h04:00: transfers happened, 11MB/s from various sources,~12h to be completed
  • quantity_rates_CSCS_20090926-24h.png

quality_CSCS_20090926-24h.png

[ phedex@cmsvobox ~]$ /home/phedex/PHEDEX/Utilities/InspectPhedexLog -es "2009-09-25 12:00:00" -t "2009-09-26 06:00:00" download.1 
given starttime 2009-09-25 10:00:00                                                                                               
given endtime 2009-09-26 04:00:00                                                                                                 


==============
ERROR ANALYSIS
==============


Data base Errors
==================


Expired tasks
==================
Total: 5          


Error message statistics per site:
===================================

 *** ERRORS from T1_IT_CNAF_Buffer:***
      1   SOURCE error during TRANSFER_PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 300 seconds

 *** ERRORS from T1_ES_PIC_Buffer:***
      7   DESTINATION error during TRANSFER_PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare Destination file in 180 seconds
      4   DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] failed to contact on remote SRM [httpg://storage01.lcg.cscs.ch:8443/srm/managerv2]. Givin' up after 3 tries

 *** ERRORS from T1_TW_ASGC_Buffer:***
    158   SOURCE error during PREPARATION phase: [GENERAL_FAILURE] Misconfiguration: No default service classes defined
      4   transfer expired in the PhEDEx download agent queue after [hours] h

 *** ERRORS from T1_US_FNAL_Buffer:***
     21   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
     20   Could not submit to FTS ~~

SITE STATISTICS:
==================
                         first entry: 2009-09-25 11:52:57      last entry: 2009-09-26 03:36:36
    T1_ES_PIC_Buffer (OK: 191  Err:  11  Exp:   0  Canc:   0  Lost:   0)   succ.: 94.6 %   total:  189.5 GB  ( 3.3 MB/s)
T1_FR_CCIN2P3_Buffer (OK:   1  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:    1.0 GB  ( 0.0 MB/s)
   T1_IT_CNAF_Buffer (OK: 148  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 99.3 %   total:  145.7 GB  ( 2.6 MB/s)
   T1_TW_ASGC_Buffer (OK: 315  Err: 158  Exp:   4  Canc:   0  Lost:   0)   succ.: 66.6 %   total:  311.9 GB  ( 5.5 MB/s)
   T1_US_FNAL_Buffer (OK: 261  Err:  41  Exp:   0  Canc:   0  Lost:   0)   succ.: 86.4 %   total:  386.9 GB  ( 6.8 MB/s)

TOTAL SUMMARY:
==================
                         first entry: 2009-09-25 11:52:57      last entry: 2009-09-26 03:36:36
total transferred: 963.8 GB  in 15.7 hours
avg. total rate: 17.4 MB/s = 139.5 Mb/s  = 1470.8 GB/day
 

Opened ticket for errors in ASGC: https://savannah.cern.ch/support/index.php?110207

29. 09. 2009 Retransfer of ALL missing blocks of Summer09 datasets

  • h 10:50: subscription of all the remaining blocks
  • ~h 11:00:transfers start, after deletions
  • h 15:50: subscribed and approved JobRobot lost blocks:

 /QCD_Pt_80_120/CMSSW_1_6_7-JobRobot-1201523639/GEN-SIM-DIGI-RECO#a1608ef9-6c0c-46e7-ae32-f4f99af21b39
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#1c34ef97-a4ff-4b6c-a5e1-401d3ec3a19c
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#272e6c35-435e-4fe9-9108-980761414eb8
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#8fcc5e95-583b-4658-8436-5d7addb27de6
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#9a26703c-acce-453b-a900-5c9782826831
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#a081486b-0296-46b4-9ca7-b1aba278407a
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#bca6223c-f060-4037-a36b-5761a8dec983
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#c15f9d45-da8a-46cd-af9f-b8873530711a
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#f477553f-2b34-4ca7-9659-663e59c80610
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#fe385fa5-5c51-4361-937f-ab164cfc7167

  • 30/09/2009, h 09:45: approved AnalysisOps requests 63359 (req6) 63419 (req 9) and 64459 (afterglow). Set to High Priority

  • 04/10/2009, ~6:00 am: transfers finished, for a total of ~16TB in ~5d

  • CSCS_quantity_cumulative_20090928-20091004.png:
    CSCS_quantity_cumulative_20090928-20091004.png

  • CSCS_quantity_rates_20090928-20091004.png:
    CSCS_quantity_rates_20090928-20091004.png

  • CSCS_quality_all_20090928-20091004.png:
    CSCS_quality_all_20090928-20091004.png

Arrow left Go to previous page / next page of CMS site log MOVED TO...

-- LeonardoSala - 2009-09-25


Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng CSCS_quality_all_20090928-20091004.png r1 manage 55.4 K 2009-10-07 - 15:04 LeonardoSala  
PNGpng CSCS_quantity_cumulative_20090928-20091004.png r1 manage 56.6 K 2009-10-07 - 15:04 LeonardoSala  
PNGpng CSCS_quantity_rates_20090928-20091004.png r1 manage 52.9 K 2009-10-07 - 15:04 LeonardoSala  
Unknown file formatlist lostBlocks_summer09.list r1 manage 29.8 K 2009-09-25 - 10:26 LeonardoSala list of lost Summer09 datasets blocks
PNGpng quality_CSCS_20090926-24h.png r1 manage 36.8 K 2009-09-29 - 08:52 LeonardoSala  
PNGpng quantity_rates_CSCS_20090926-24h.png r1 manage 34.1 K 2009-09-29 - 08:31 LeonardoSala  
Edit | Attach | Watch | Print version | History: r6 < r5 < r4 < r3 < r2 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r6 - 2009-10-07 - LeonardoSala
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback