CMSSiteLog20 < LCGTier2

Tags: view all tags

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

%TOC%

%ICON{arrowleft}% Go to [[CMSSiteLogXX][previous page]] / [[CMSSiteLogXX][next page]] of CMS site log %M%

---+ 25. 09. 2009 Retransfer of missing blocks of Summer09 datasets

List of lost blocks: [[%ATTACHURL%/lostBlocks_summer09.list][lostBlocks_summer09.list]]: list of lost Summer09 datasets blocks

   * h 11:42: Requested first part of lost Summer09 lost blocks, size ~1TB, for testing purposes. Request 62359:

https://cmsweb.cern.ch/phedex/prod/Request::View?request=62359&.submit=Submit

   * h 12:00: Prod Agents stopped in order to manually clear lost pnsfs, as they are hanging jobs on dCache
   * h 14:00 - 26/09/2009 h04:00: transfers happened, 11MB/s from various sources,~12h to be completed
   * <img width="800" alt="quantity_rates_CSCS_20090926-24h.png" src="https://twiki.cscs.ch/twiki/pub/LCGTier2/CMSSiteLog20/quantity_rates_CSCS_20090926-24h.png" height="500" />

---+ 

<img width="800" alt="quality_CSCS_20090926-24h.png" src="https://twiki.cscs.ch/twiki/pub/LCGTier2/CMSSiteLog20/quality_CSCS_20090926-24h.png" height="620" />

<verbatim>[ phedex@cmsvobox ~]$ /home/phedex/PHEDEX/Utilities/InspectPhedexLog -es "2009-09-25 12:00:00" -t "2009-09-26 06:00:00" download.1 
given starttime 2009-09-25 10:00:00                                                                                               
given endtime 2009-09-26 04:00:00                                                                                                 


==============
ERROR ANALYSIS
==============


Data base Errors
==================


Expired tasks
==================
Total: 5          


Error message statistics per site:
===================================

 *** ERRORS from T1_IT_CNAF_Buffer:***
      1   SOURCE error during TRANSFER_PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 300 seconds

 *** ERRORS from T1_ES_PIC_Buffer:***
      7   DESTINATION error during TRANSFER_PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare Destination file in 180 seconds
      4   DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] failed to contact on remote SRM [httpg://storage01.lcg.cscs.ch:8443/srm/managerv2]. Givin' up after 3 tries

 *** ERRORS from T1_TW_ASGC_Buffer:***
    158   SOURCE error during PREPARATION phase: [GENERAL_FAILURE] Misconfiguration: No default service classes defined
      4   transfer expired in the PhEDEx download agent queue after [hours] h

 *** ERRORS from T1_US_FNAL_Buffer:***
     21   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
     20   Could not submit to FTS ~~

SITE STATISTICS:
==================
                         first entry: 2009-09-25 11:52:57      last entry: 2009-09-26 03:36:36
    T1_ES_PIC_Buffer (OK: 191  Err:  11  Exp:   0  Canc:   0  Lost:   0)   succ.: 94.6 %   total:  189.5 GB  ( 3.3 MB/s)
T1_FR_CCIN2P3_Buffer (OK:   1  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:    1.0 GB  ( 0.0 MB/s)
   T1_IT_CNAF_Buffer (OK: 148  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 99.3 %   total:  145.7 GB  ( 2.6 MB/s)
   T1_TW_ASGC_Buffer (OK: 315  Err: 158  Exp:   4  Canc:   0  Lost:   0)   succ.: 66.6 %   total:  311.9 GB  ( 5.5 MB/s)
   T1_US_FNAL_Buffer (OK: 261  Err:  41  Exp:   0  Canc:   0  Lost:   0)   succ.: 86.4 %   total:  386.9 GB  ( 6.8 MB/s)

TOTAL SUMMARY:
==================
                         first entry: 2009-09-25 11:52:57      last entry: 2009-09-26 03:36:36
total transferred: 963.8 GB  in 15.7 hours
avg. total rate: 17.4 MB/s = 139.5 Mb/s  = 1470.8 GB/day
 </verbatim>

---+ 

Opened ticket for errors in ASGC: https://savannah.cern.ch/support/index.php?110207

---+ 

---+ 29. 09. 2009 Retransfer of ALL missing blocks of Summer09 datasets

   * h 10:50: subscription of all the remaining blocks
   * ~h 11:00:transfers start, after deletions
   * h 15:50: subscribed and approved JobRobot lost blocks:

<verbatim> /QCD_Pt_80_120/CMSSW_1_6_7-JobRobot-1201523639/GEN-SIM-DIGI-RECO#a1608ef9-6c0c-46e7-ae32-f4f99af21b39
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#1c34ef97-a4ff-4b6c-a5e1-401d3ec3a19c
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#272e6c35-435e-4fe9-9108-980761414eb8
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#8fcc5e95-583b-4658-8436-5d7addb27de6
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#9a26703c-acce-453b-a900-5c9782826831
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#a081486b-0296-46b4-9ca7-b1aba278407a
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#bca6223c-f060-4037-a36b-5761a8dec983
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#c15f9d45-da8a-46cd-af9f-b8873530711a
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#f477553f-2b34-4ca7-9659-663e59c80610
/QCD_pt_0_15/JobRobot_IDEAL_V9_JobRobot/GEN-SIM-RAW-RECO#fe385fa5-5c51-4361-937f-ab164cfc7167</verbatim>

   * 30/09/2009, h 09:45: approved AnalysisOps requests 63359 (req6) 63419 (req 9) and 64459 (afterglow). Set to High Priority

   * 04/10/2009, ~6:00 am: transfers finished, for a total of ~16TB in ~5d

   * CSCS_quantity_cumulative_20090928-20091004.png: <br /> <img width="800" alt="CSCS_quantity_cumulative_20090928-20091004.png" src="/twiki/pub/LCGTier2/CMSSiteLog20/CSCS_quantity_cumulative_20090928-20091004.png" height="500" />

   * CSCS_quantity_rates_20090928-20091004.png: <br /> <img width="800" alt="CSCS_quantity_rates_20090928-20091004.png" src="/twiki/pub/LCGTier2/CMSSiteLog20/CSCS_quantity_rates_20090928-20091004.png" height="500" />

   * CSCS_quality_all_20090928-20091004.png: <br /> <img width="800" alt="CSCS_quality_all_20090928-20091004.png" src="/twiki/pub/LCGTier2/CMSSiteLog20/CSCS_quality_all_20090928-20091004.png" height="620" />

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

%ICON{arrowleft}% Go to [[CMSSiteLogXX][previous page]] / [[CMSSiteLogXX][next page]] of CMS site log %M%

-- Main.LeonardoSala - 2009-09-25

---

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	CSCS_quality_all_20090928-20091004.png	r1	manage	55.4 K	2009-10-07 - 15:04	LeonardoSala
png	CSCS_quantity_cumulative_20090928-20091004.png	r1	manage	56.6 K	2009-10-07 - 15:04	LeonardoSala
png	CSCS_quantity_rates_20090928-20091004.png	r1	manage	52.9 K	2009-10-07 - 15:04	LeonardoSala
list	lostBlocks_summer09.list	r1	manage	29.8 K	2009-09-25 - 10:26	LeonardoSala	list of lost Summer09 datasets blocks
png	quality_CSCS_20090926-24h.png	r1	manage	36.8 K	2009-09-29 - 08:52	LeonardoSala
png	quantity_rates_CSCS_20090926-24h.png	r1	manage	34.1 K	2009-09-29 - 08:31	LeonardoSala