Tags:
create new tag
view all tags

Arrow left Go to previous page / next page of CMS site log MOVED TO...

19. 09. 2009 Steady PhEDEx Throughput of >100 MB/s over last 12 hours while rearranging pools!

Even though we were doing multiple pool migrations over the fileservers (for converting all file servers to raidz2), the infrastructure managed to keep up a high and quite reliable throughput over the last 12 hours.

generated on Sat Sep 19 09:00:01 CEST 2009 
------------------------
given starttime 2009-09-18 19:00:05
given endtime 2009-09-19 07:00:05

ERROR ANALYSIS Show Hide



Data base Errors
==================


Expired tasks
==================
Total: 7


Error message statistics per site:
===================================

 *** ERRORS from T1_DE_FZK_Buffer:***
     74   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer was killed)
      1   error during phase: [] 

 *** ERRORS from T1_IT_CNAF_Buffer:***
      3   DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt300/AODSIM/MC_31X_V3_AODSIM-v1/0016
      1   DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt300/AODSIM/MC_31X_V3_AODSIM-v1/0016 not found ( .(id)(0016) )

 *** ERRORS from T1_FR_CCIN2P3_Buffer:***
     13   SOURCE error during TRANSFER_PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 180 seconds

 *** ERRORS from T2_DE_RWTH:***
      7   transfer expired in the PhEDEx download agent queue after [hours] h

 *** ERRORS from T1_US_FNAL_Buffer:***
      3   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)

SITE STATISTICS:
==================
                         first entry: 2009-09-18 19:00:16      last entry: 2009-09-19 06:59:54
   T1_CH_CERN_Buffer (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
    T1_DE_FZK_Buffer (OK: 1754  Err:  75  Exp:   0  Canc:   0  Lost:   0)   succ.: 95.9 %   total: 2724.6 GB  (63.1 MB/s)
    T1_ES_PIC_Buffer (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
T1_FR_CCIN2P3_Buffer (OK: 217  Err:  13  Exp:   0  Canc:   0  Lost:   0)   succ.: 94.3 %   total:  234.3 GB  ( 5.4 MB/s)
   T1_IT_CNAF_Buffer (OK: 984  Err:   4  Exp:   0  Canc:   0  Lost:   0)   succ.: 99.6 %   total: 1526.1 GB  (35.3 MB/s)
   T1_TW_ASGC_Buffer (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   22.5 GB  ( 0.5 MB/s)
    T1_UK_RAL_Buffer (OK:   9  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   23.4 GB  ( 0.5 MB/s)
   T1_US_FNAL_Buffer (OK:  25  Err:   3  Exp:   0  Canc:   0  Lost:   0)   succ.: 89.3 %   total:   68.7 GB  ( 1.6 MB/s)
          T2_DE_RWTH (OK:   6  Err:   0  Exp:   7  Canc:   0  Lost:   0)   succ.: 100.0 %   total:    6.1 GB  ( 0.1 MB/s)

TOTAL SUMMARY:
==================
                         first entry: 2009-09-18 19:00:16      last entry: 2009-09-19 06:59:54
total transferred: 4329.4 GB  in 12.0 hours
avg. total rate: 102.7 MB/s = 821.4 Mb/s  = 8663.1 GB/day

Central monitoring plots

Rate plot for the last 12h

phedex-rateplot-20090919.jpg

Note: The activitiy table is for the last 24h

phedex-activity-thrpt.jpg

Plot of the requested volume for the last 72h. The sharp rise corresponds to a number of B-physics datasets ordered for Urs Langenegger

phedex-requested-20090920.jpg

Tailing by one data set

One data set is responsible for the flattening out of the requested volume plot, above, as can be seen from the subscriptions table: /ppEleX/Summer09-MC_31X_V3-v1/GEN-SIM-RECO. Note that all the sets listed in this subscriptions table were ordered at the same time on 2009-09-17.

phedex-subscriptions-20090922.jpg

The reason for this corresponds to the low transfer throughput from T1_FR_CCIN2P3. All current transfers to our site for this set emerge from T1_FR_CCIN2P3.

phedex-state-details-20090921.jpg

Local monitoring plots

From the monitoring plots it surprisingly becomes clear that this feat is accomplished by a limited amount of WAN movers (always <10). A lot of the active pools have set their max_movers to 2 only, while the others get queued. Investigating why the movers have been limited in that way...

queued_movers_wan-20090919-0918-day.gif movers_wan-20090919-0918-day.gif movers_default-20090919-0918-day.gif fileservers-nw-report-20090919-0918-day.gif

-- DerekFeichtinger - 2009-09-19

Arrow left Go to previous page / next page of CMS site log MOVED TO...

Topic attachments
I Attachment HistorySorted ascending Action Size Date Who Comment
GIFgif fileservers-nw-report-20090919-0918-day.gif r1 manage 18.8 K 2009-09-19 - 07:33 DerekFeichtinger  
GIFgif movers_default-20090919-0918-day.gif r1 manage 14.9 K 2009-09-19 - 07:33 DerekFeichtinger  
GIFgif movers_wan-20090919-0918-day.gif r1 manage 17.3 K 2009-09-19 - 07:32 DerekFeichtinger  
JPEGjpg phedex-activity-thrpt.jpg r1 manage 41.3 K 2009-09-19 - 07:54 DerekFeichtinger  
JPEGjpg phedex-rateplot-20090919.jpg r1 manage 53.6 K 2009-09-19 - 07:58 DerekFeichtinger  
JPEGjpg phedex-requested-20090920.jpg r1 manage 59.3 K 2009-09-19 - 08:08 DerekFeichtinger  
JPEGjpg phedex-state-details-20090921.jpg r1 manage 26.6 K 2009-09-19 - 08:23 DerekFeichtinger  
JPEGjpg phedex-subscriptions-20090922.jpg r1 manage 82.3 K 2009-09-19 - 08:28 DerekFeichtinger  
GIFgif queued_movers_wan-20090919-0918-day.gif r1 manage 18.8 K 2009-09-19 - 07:31 DerekFeichtinger  
Topic revision: r1 - 2009-09-19 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback