Go to
previous page /
next page of CMS site log
19. 09. 2009 Steady PhEDEx Throughput of >100 MB/s over last 12 hours while rearranging pools!
Even though we were doing multiple pool migrations over the fileservers (for converting all file servers to raidz2), the infrastructure managed to keep up a high and quite reliable throughput over the last 12 hours.
generated on Sat Sep 19 09:00:01 CEST 2009
------------------------
given starttime 2009-09-18 19:00:05
given endtime 2009-09-19 07:00:05
ERROR ANALYSIS
Show Hide
Data base Errors
==================
Expired tasks
==================
Total: 7
Error message statistics per site:
===================================
*** ERRORS from T1_DE_FZK_Buffer:***
74 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer was killed)
1 error during phase: []
*** ERRORS from T1_IT_CNAF_Buffer:***
3 DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt300/AODSIM/MC_31X_V3_AODSIM-v1/0016
1 DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt300/AODSIM/MC_31X_V3_AODSIM-v1/0016 not found ( .(id)(0016) )
*** ERRORS from T1_FR_CCIN2P3_Buffer:***
13 SOURCE error during TRANSFER_PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds
*** ERRORS from T2_DE_RWTH:***
7 transfer expired in the PhEDEx download agent queue after [hours] h
*** ERRORS from T1_US_FNAL_Buffer:***
3 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
SITE STATISTICS:
==================
first entry: 2009-09-18 19:00:16 last entry: 2009-09-19 06:59:54
T1_CH_CERN_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T1_DE_FZK_Buffer (OK: 1754 Err: 75 Exp: 0 Canc: 0 Lost: 0) succ.: 95.9 % total: 2724.6 GB (63.1 MB/s)
T1_ES_PIC_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T1_FR_CCIN2P3_Buffer (OK: 217 Err: 13 Exp: 0 Canc: 0 Lost: 0) succ.: 94.3 % total: 234.3 GB ( 5.4 MB/s)
T1_IT_CNAF_Buffer (OK: 984 Err: 4 Exp: 0 Canc: 0 Lost: 0) succ.: 99.6 % total: 1526.1 GB (35.3 MB/s)
T1_TW_ASGC_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 22.5 GB ( 0.5 MB/s)
T1_UK_RAL_Buffer (OK: 9 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 23.4 GB ( 0.5 MB/s)
T1_US_FNAL_Buffer (OK: 25 Err: 3 Exp: 0 Canc: 0 Lost: 0) succ.: 89.3 % total: 68.7 GB ( 1.6 MB/s)
T2_DE_RWTH (OK: 6 Err: 0 Exp: 7 Canc: 0 Lost: 0) succ.: 100.0 % total: 6.1 GB ( 0.1 MB/s)
TOTAL SUMMARY:
==================
first entry: 2009-09-18 19:00:16 last entry: 2009-09-19 06:59:54
total transferred: 4329.4 GB in 12.0 hours
avg. total rate: 102.7 MB/s = 821.4 Mb/s = 8663.1 GB/day
Central monitoring plots
Rate plot for the last 12h
Note: The activitiy table is for the last 24h
Plot of the requested volume for the last 72h. The sharp rise corresponds to a number of B-physics datasets ordered for Urs Langenegger
Tailing by one data set
One data set is responsible for the flattening out of the
requested volume plot, above, as can be seen from the subscriptions table:
/ppEleX/Summer09-MC_31X_V3-v1/GEN-SIM-RECO. Note that all the sets listed in this subscriptions table were ordered at the same time on 2009-09-17.
The reason for this corresponds to the low transfer throughput from T1_FR_CCIN2P3. All current transfers to our site for this set emerge from T1_FR_CCIN2P3.
Local monitoring plots
From the monitoring plots it surprisingly becomes clear that this feat is accomplished by a limited amount of WAN movers (always <10). A lot of the active pools have set their max_movers to 2 only, while the others get queued. Investigating why the movers have been limited in that way...
--
DerekFeichtinger - 2009-09-19
Go to
previous page /
next page of CMS site log