Go to
previous page /
next page of CMS site log
31. 08. 2009 Good transfer quality on first Analysis Operations transfers
/PhotonJet_Pt30/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/InclusiveMu5_Pt350/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt3000/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt80/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/InclusiveMu5_Pt150/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt80/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt300/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt3000/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/Wmunu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt30/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/Zmumu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt470/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt470/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt300/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt15/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt1400/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/Ztautau/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt1400/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt2200/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt15/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt800/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt2200/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/Zee/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt800/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/QCD_Pt170/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/InclusiveMu5_Pt50/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/Wtaunu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/InclusiveMu5_Pt250/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/Wenu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
/PhotonJet_Pt170/Summer09-MC_31X_V3_AODSIM-v1/AODSIM
generated on Mon Aug 31 17:15:01 CEST 2009
------------------------
given starttime 2009-08-31 03:15:03
given endtime 2009-08-31 15:15:03
==============
ERROR ANALYSIS
==============
Data base Errors
==================
Expired tasks
==================
Total: 0
Error message statistics per site:
===================================
*** ERRORS from T1_DE_FZK_Buffer:***
4 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer was killed)
*** ERRORS from T1_ES_PIC_Buffer:***
1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Broken pipe)
*** ERRORS from T1_CH_CERN_Buffer:***
1 SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] Timed out
*** ERRORS from T1_TW_ASGC_Buffer:***
16 DESTINATION error during PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014 reason: File exists
5 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014
2 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0017
1 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/Wenu/AODSIM/MC_31X_V3_AODSIM-v1/0020
1 DESTINATION error during PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021 reason: File exists
1 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021
1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0017 not found ( .(id)(0017) )
1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/Wenu/AODSIM/MC_31X_V3_AODSIM-v1/0020 not found ( .(id)(0020) )
1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014 not found ( .(id)(0014) )
1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021 not found ( .(id)(0021) )
*** ERRORS from T1_US_FNAL_Buffer:***
32 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
6 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0021 reason: File exists
5 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_xio: Unable to connect to cmsstor55.fnal.gov:2811 globus_xio: System error in connect: Connection refused globus_xio: A system call failed: Connection refused
4 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
1 SOURCE error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] [SrmPing] failed: SOAP-ENV:Client - CGSI-gSOAP running on cmsfts1.fnal.gov reports Could NOT load client credentials GSS Major Status: General failure GSS Minor Status Error Chain: globus_gsi_gssapi: Error with GSI credential globus_gsi_gssapi: Error with gss credential handle globus_...(error cut)
1 DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0026
1 DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0026 not found ( .(id)(0026) )
1 DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0021 not found ( .(id)(0021) )
1 SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out
*** ERRORS from T1_UK_RAL_Buffer:***
4 DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0008
1 DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0008 not found ( .(id)(0008) )
SITE STATISTICS:
==================
first entry: 2009-08-31 03:15:16 last entry: 2009-08-31 15:14:55
T1_CH_CERN_Buffer (OK: 8 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 88.9 % total: 21.5 GB ( 0.5 MB/s)
T1_DE_FZK_Buffer (OK: 295 Err: 4 Exp: 0 Canc: 0 Lost: 0) succ.: 98.7 % total: 298.5 GB ( 6.9 MB/s)
T1_ES_PIC_Buffer (OK: 290 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 99.7 % total: 336.0 GB ( 7.8 MB/s)
T1_FR_CCIN2P3_Buffer (OK: 165 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 165.1 GB ( 3.8 MB/s)
T1_IT_CNAF_Buffer (OK: 131 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 141.7 GB ( 3.3 MB/s)
T1_TW_ASGC_Buffer (OK: 518 Err: 30 Exp: 0 Canc: 0 Lost: 0) succ.: 94.5 % total: 509.2 GB (11.8 MB/s)
T1_UK_RAL_Buffer (OK: 510 Err: 5 Exp: 0 Canc: 0 Lost: 0) succ.: 99.0 % total: 521.0 GB (12.1 MB/s)
T1_US_FNAL_Buffer (OK: 758 Err: 52 Exp: 0 Canc: 0 Lost: 0) succ.: 93.6 % total: 1171.8 GB (27.1 MB/s)
T2_DE_RWTH (OK: 178 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 430.5 GB (10.0 MB/s)
TOTAL SUMMARY:
==================
first entry: 2009-08-31 03:15:16 last entry: 2009-08-31 15:14:55
total transferred: 3348.4 GB in 12.0 hours
avg. total rate: 79.4 MB/s = 635.3 Mb/s = 6700.0 GB/day
1. 09. 2009 Major dcache Problems, recovery on the afternoon. Error reasons unclear
Experienced strange dcache problems since yesterday 17:30. All transfers failed. Messages seem to point to communication problems between cells. CSCS was not able to restart the service (SRM cell would not start). After executing the deploy_srmv2.sh, they were able to start it again, but seemingly the thing still failed.
This was unfortunately overlaid by a cron job for repairing the now more often seen root owned directories. The cron job erroneously modified the top layer directories to belong to root:bin, thereby failing all our transfers.
In the afternoon the system seemed to have recovered as far as we could tell, but we still do not understand all the reasons for the different errors. Decided to migrate to chimera the next day, if the system proofs to be stable in the next night (so that at least we can assume that the DB is not hopelessly corrupted).
2. 09. 2009 Again, Good transfer quality
generated on Wed Sep 2 08:45:01 CEST 2009
------------------------
given starttime 2009-09-01 18:45:03
given endtime 2009-09-02 06:45:03
==============
ERROR ANALYSIS
==============
Data base Errors
==================
Expired tasks
==================
Total: 0
Error message statistics per site:
===================================
*** ERRORS from T1_TW_ASGC_Buffer:***
1 DESTINATION error during PREPARATION phase: [CONNECTION] [srm2__srmPrepareToPut] failed: SOAP-ENV:Client - CGSI-gSOAP: Could not open connection !
1 SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 300 seconds
*** ERRORS from T1_US_FNAL_Buffer:***
96 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
27 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/ppMuX/GEN-SIM-RECO/MC_31X_V3-v1/0004 reason: File exists
2 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path
2 SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] FAILED: at [date] state Failed : ThePinCallbacks Timeout
1 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Stream ended before EOD/Data channel from /131.225.189.76:33581 was closed before EOD marker)
1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Stream ended before EOD/Data channel from /131.225.189.6:57905 was closed before EOD marker)
1 SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] FAILED: at [date] state Failed : No Route to cell for packet {uoid=stuff}
1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
SITE STATISTICS:
==================
first entry: 2009-09-01 18:45:06 last entry: 2009-09-02 06:45:00
T1_CH_CERN_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T1_DE_FZK_Buffer (OK: 1 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 2.5 GB ( 0.1 MB/s)
T1_ES_PIC_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T1_FR_CCIN2P3_Buffer (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 24.7 GB ( 0.6 MB/s)
T1_IT_CNAF_Buffer (OK: 88 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 101.8 GB ( 2.4 MB/s)
T1_TW_ASGC_Buffer (OK: 12 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 85.7 % total: 24.6 GB ( 0.6 MB/s)
T1_UK_RAL_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 22.2 GB ( 0.5 MB/s)
T1_US_FNAL_Buffer (OK: 2211 Err: 132 Exp: 0 Canc: 0 Lost: 0) succ.: 94.4 % total: 3486.0 GB (80.7 MB/s)
T2_DE_RWTH (OK: 180 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 435.4 GB (10.1 MB/s)
TOTAL SUMMARY:
==================
first entry: 2009-09-01 18:45:06 last entry: 2009-09-02 06:45:00
total transferred: 3856.0 GB in 12.0 hours
avg. total rate: 91.4 MB/s = 731.3 Mb/s = 7713.0 GB/day
--
DerekFeichtinger - 2009-08-31
Go to
previous page /
next page of CMS site log