Tags:
create new tag
view all tags

Arrow left Go to previous page / next page of CMS site log MOVED TO...

31. 08. 2009 Good transfer quality on first Analysis Operations transfers

     /PhotonJet_Pt30/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /InclusiveMu5_Pt350/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt3000/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt80/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /InclusiveMu5_Pt150/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt80/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt300/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt3000/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /Wmunu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt30/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /Zmumu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt470/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt470/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt300/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt15/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt1400/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /Ztautau/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt1400/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt2200/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt15/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt800/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt2200/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /Zee/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt800/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /QCD_Pt170/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /InclusiveMu5_Pt50/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /Wtaunu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /InclusiveMu5_Pt250/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /Wenu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
     /PhotonJet_Pt170/Summer09-MC_31X_V3_AODSIM-v1/AODSIM

generated on Mon Aug 31 17:15:01 CEST 2009 
------------------------
given starttime 2009-08-31 03:15:03
given endtime 2009-08-31 15:15:03


==============
ERROR ANALYSIS
==============


Data base Errors
==================


Expired tasks
==================
Total: 0


Error message statistics per site:
===================================

 *** ERRORS from T1_DE_FZK_Buffer:***
      4   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer was killed)

 *** ERRORS from T1_ES_PIC_Buffer:***
      1   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Broken pipe)

 *** ERRORS from T1_CH_CERN_Buffer:***
      1   SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] Timed out

 *** ERRORS from T1_TW_ASGC_Buffer:***
     16   DESTINATION error during PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014 reason: File exists
      5   DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014
      2   DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0017
      1   DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/Wenu/AODSIM/MC_31X_V3_AODSIM-v1/0020
      1   DESTINATION error during PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021 reason: File exists
      1   DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021
      1   DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0017 not found ( .(id)(0017) )
      1   DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/Wenu/AODSIM/MC_31X_V3_AODSIM-v1/0020 not found ( .(id)(0020) )
      1   DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014 not found ( .(id)(0014) )
      1   DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021 not found ( .(id)(0021) )

 *** ERRORS from T1_US_FNAL_Buffer:***
     32   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
      6   DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0021 reason: File exists
      5   SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_xio: Unable to connect to cmsstor55.fnal.gov:2811 globus_xio: System error in connect: Connection refused globus_xio: A system call failed: Connection refused
      4   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
      1   SOURCE error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] [SrmPing] failed: SOAP-ENV:Client - CGSI-gSOAP running on cmsfts1.fnal.gov reports Could NOT load client credentials GSS Major Status: General failure GSS Minor Status Error Chain: globus_gsi_gssapi: Error with GSI credential globus_gsi_gssapi: Error with gss credential handle globus_...(error cut)
      1   DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0026
      1   DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0026 not found ( .(id)(0026) )
      1   DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0021 not found ( .(id)(0021) )
      1   SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out

 *** ERRORS from T1_UK_RAL_Buffer:***
      4   DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0008
      1   DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0008 not found ( .(id)(0008) )

SITE STATISTICS:
==================
                         first entry: 2009-08-31 03:15:16      last entry: 2009-08-31 15:14:55
   T1_CH_CERN_Buffer (OK:   8  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 88.9 %   total:   21.5 GB  ( 0.5 MB/s)
    T1_DE_FZK_Buffer (OK: 295  Err:   4  Exp:   0  Canc:   0  Lost:   0)   succ.: 98.7 %   total:  298.5 GB  ( 6.9 MB/s)
    T1_ES_PIC_Buffer (OK: 290  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 99.7 %   total:  336.0 GB  ( 7.8 MB/s)
T1_FR_CCIN2P3_Buffer (OK: 165  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  165.1 GB  ( 3.8 MB/s)
   T1_IT_CNAF_Buffer (OK: 131  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  141.7 GB  ( 3.3 MB/s)
   T1_TW_ASGC_Buffer (OK: 518  Err:  30  Exp:   0  Canc:   0  Lost:   0)   succ.: 94.5 %   total:  509.2 GB  (11.8 MB/s)
    T1_UK_RAL_Buffer (OK: 510  Err:   5  Exp:   0  Canc:   0  Lost:   0)   succ.: 99.0 %   total:  521.0 GB  (12.1 MB/s)
   T1_US_FNAL_Buffer (OK: 758  Err:  52  Exp:   0  Canc:   0  Lost:   0)   succ.: 93.6 %   total: 1171.8 GB  (27.1 MB/s)
          T2_DE_RWTH (OK: 178  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  430.5 GB  (10.0 MB/s)

TOTAL SUMMARY:
==================
                         first entry: 2009-08-31 03:15:16      last entry: 2009-08-31 15:14:55
total transferred: 3348.4 GB  in 12.0 hours
avg. total rate: 79.4 MB/s = 635.3 Mb/s  = 6700.0 GB/day

1. 09. 2009 Major dcache Problems, recovery on the afternoon. Error reasons unclear

Experienced strange dcache problems since yesterday 17:30. All transfers failed. Messages seem to point to communication problems between cells. CSCS was not able to restart the service (SRM cell would not start). After executing the deploy_srmv2.sh, they were able to start it again, but seemingly the thing still failed. This was unfortunately overlaid by a cron job for repairing the now more often seen root owned directories. The cron job erroneously modified the top layer directories to belong to root:bin, thereby failing all our transfers. In the afternoon the system seemed to have recovered as far as we could tell, but we still do not understand all the reasons for the different errors. Decided to migrate to chimera the next day, if the system proofs to be stable in the next night (so that at least we can assume that the DB is not hopelessly corrupted).

2. 09. 2009 Again, Good transfer quality

generated on Wed Sep 2 08:45:01 CEST 2009 
------------------------
given starttime 2009-09-01 18:45:03
given endtime 2009-09-02 06:45:03


==============
ERROR ANALYSIS
==============


Data base Errors
==================


Expired tasks
==================
Total: 0


Error message statistics per site:
===================================

 *** ERRORS from T1_TW_ASGC_Buffer:***
      1   DESTINATION error during PREPARATION phase: [CONNECTION] [srm2__srmPrepareToPut] failed: SOAP-ENV:Client - CGSI-gSOAP: Could not open connection !
      1   SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 300 seconds

 *** ERRORS from T1_US_FNAL_Buffer:***
     96   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
     27   DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/ppMuX/GEN-SIM-RECO/MC_31X_V3-v1/0004 reason: File exists
      2   DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path
      2   SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] FAILED: at [date] state Failed : ThePinCallbacks Timeout
      1   SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
      1   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Stream ended before EOD/Data channel from /131.225.189.76:33581 was closed before EOD marker)
      1   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Stream ended before EOD/Data channel from /131.225.189.6:57905 was closed before EOD marker)
      1   SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] FAILED: at [date] state Failed : No Route to cell for packet {uoid=stuff}
      1   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out

SITE STATISTICS:
==================
                         first entry: 2009-09-01 18:45:06      last entry: 2009-09-02 06:45:00
   T1_CH_CERN_Buffer (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
    T1_DE_FZK_Buffer (OK:   1  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:    2.5 GB  ( 0.1 MB/s)
    T1_ES_PIC_Buffer (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
T1_FR_CCIN2P3_Buffer (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   24.7 GB  ( 0.6 MB/s)
   T1_IT_CNAF_Buffer (OK:  88  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  101.8 GB  ( 2.4 MB/s)
   T1_TW_ASGC_Buffer (OK:  12  Err:   2  Exp:   0  Canc:   0  Lost:   0)   succ.: 85.7 %   total:   24.6 GB  ( 0.6 MB/s)
    T1_UK_RAL_Buffer (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   22.2 GB  ( 0.5 MB/s)
   T1_US_FNAL_Buffer (OK: 2211  Err: 132  Exp:   0  Canc:   0  Lost:   0)   succ.: 94.4 %   total: 3486.0 GB  (80.7 MB/s)
          T2_DE_RWTH (OK: 180  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  435.4 GB  (10.1 MB/s)

TOTAL SUMMARY:
==================
                         first entry: 2009-09-01 18:45:06      last entry: 2009-09-02 06:45:00
total transferred: 3856.0 GB  in 12.0 hours
avg. total rate: 91.4 MB/s = 731.3 Mb/s  = 7713.0 GB/day

-- DerekFeichtinger - 2009-08-31


Arrow left Go to previous page / next page of CMS site log MOVED TO...

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2009-09-17 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback