Tags:
tag this topic
create new tag
view all tags
<!-- keep this as a security measure: * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup #uncomment this if you want the page only be viewable by the internal people # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup --> %TOC% %ICON{arrowleft}% Go to [[CMSSiteLog15][previous page]] / [[CMSSiteLog17][next page]] of CMS site log %M% ---+ 31. 08. 2009 Good transfer quality on first Analysis Operations transfers <pre> /PhotonJet_Pt30/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /InclusiveMu5_Pt350/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt3000/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt80/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /InclusiveMu5_Pt150/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt80/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt300/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt3000/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /Wmunu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt30/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /Zmumu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt470/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt470/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt300/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt15/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt1400/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /Ztautau/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt1400/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt2200/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt15/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt800/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt2200/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /Zee/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt800/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /QCD_Pt170/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /InclusiveMu5_Pt50/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /Wtaunu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /InclusiveMu5_Pt250/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /Wenu/Summer09-MC_31X_V3_AODSIM-v1/AODSIM /PhotonJet_Pt170/Summer09-MC_31X_V3_AODSIM-v1/AODSIM </pre> <pre> generated on Mon Aug 31 17:15:01 CEST 2009 ------------------------ given starttime 2009-08-31 03:15:03 given endtime 2009-08-31 15:15:03 ============== ERROR ANALYSIS ============== Data base Errors ================== Expired tasks ================== Total: 0 Error message statistics per site: =================================== *** ERRORS from T1_DE_FZK_Buffer:*** 4 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer was killed) *** ERRORS from T1_ES_PIC_Buffer:*** 1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Broken pipe) *** ERRORS from T1_CH_CERN_Buffer:*** 1 SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] Timed out *** ERRORS from T1_TW_ASGC_Buffer:*** 16 DESTINATION error during PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014 reason: File exists 5 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014 2 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0017 1 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/Wenu/AODSIM/MC_31X_V3_AODSIM-v1/0020 1 DESTINATION error during PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021 reason: File exists 1 DESTINATION error during PREPARATION phase: [PERMISSION] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021 1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0017 not found ( .(id)(0017) ) 1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/Wenu/AODSIM/MC_31X_V3_AODSIM-v1/0020 not found ( .(id)(0020) ) 1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/PhotonJet_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0014 not found ( .(id)(0014) ) 1 DESTINATION error during PREPARATION phase: [USER] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/InclusiveMu5_Pt350/AODSIM/MC_31X_V3_AODSIM-v1/0021 not found ( .(id)(0021) ) *** ERRORS from T1_US_FNAL_Buffer:*** 32 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full) 6 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0021 reason: File exists 5 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_xio: Unable to connect to cmsstor55.fnal.gov:2811 globus_xio: System error in connect: Connection refused globus_xio: A system call failed: Connection refused 4 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out 1 SOURCE error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] [SrmPing] failed: SOAP-ENV:Client - CGSI-gSOAP running on cmsfts1.fnal.gov reports Could NOT load client credentials GSS Major Status: General failure GSS Minor Status Error Chain: globus_gsi_gssapi: Error with GSI credential globus_gsi_gssapi: Error with gss credential handle globus_...(error cut) 1 DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0026 1 DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0026 not found ( .(id)(0026) ) 1 DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt800/AODSIM/MC_31X_V3_AODSIM-v1/0021 not found ( .(id)(0021) ) 1 SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out *** ERRORS from T1_UK_RAL_Buffer:*** 4 DESTINATION error during TRANSFER_PREPARATION phase: [SECURITY_ERROR] at [date] state Failed : user has no permission to write into path /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/QCD_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0008 1 DESTINATION error during TRANSFER_PREPARATION phase: [USER_ERROR] [srm-URL] Failed to create, got error return code from pnfs: path /pnfs/fs/usr/cms/trivcat/store/mc/Summer09/QCD_Pt1400/AODSIM/MC_31X_V3_AODSIM-v1/0008 not found ( .(id)(0008) ) SITE STATISTICS: ================== first entry: 2009-08-31 03:15:16 last entry: 2009-08-31 15:14:55 T1_CH_CERN_Buffer (OK: 8 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 88.9 % total: 21.5 GB ( 0.5 MB/s) T1_DE_FZK_Buffer (OK: 295 Err: 4 Exp: 0 Canc: 0 Lost: 0) succ.: 98.7 % total: 298.5 GB ( 6.9 MB/s) T1_ES_PIC_Buffer (OK: 290 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 99.7 % total: 336.0 GB ( 7.8 MB/s) T1_FR_CCIN2P3_Buffer (OK: 165 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 165.1 GB ( 3.8 MB/s) T1_IT_CNAF_Buffer (OK: 131 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 141.7 GB ( 3.3 MB/s) T1_TW_ASGC_Buffer (OK: 518 Err: 30 Exp: 0 Canc: 0 Lost: 0) succ.: 94.5 % total: 509.2 GB (11.8 MB/s) T1_UK_RAL_Buffer (OK: 510 Err: 5 Exp: 0 Canc: 0 Lost: 0) succ.: 99.0 % total: 521.0 GB (12.1 MB/s) T1_US_FNAL_Buffer (OK: 758 Err: 52 Exp: 0 Canc: 0 Lost: 0) succ.: 93.6 % total: 1171.8 GB (27.1 MB/s) T2_DE_RWTH (OK: 178 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 430.5 GB (10.0 MB/s) TOTAL SUMMARY: ================== first entry: 2009-08-31 03:15:16 last entry: 2009-08-31 15:14:55 total transferred: 3348.4 GB in 12.0 hours avg. total rate: 79.4 MB/s = 635.3 Mb/s = 6700.0 GB/day </pre> ---+ 1. 09. 2009 Major dcache Problems, recovery on the afternoon. Error reasons unclear Experienced strange dcache problems since yesterday 17:30. All transfers failed. Messages seem to point to communication problems between cells. CSCS was not able to restart the service (SRM cell would not start). After executing the deploy_srmv2.sh, they were able to start it again, but seemingly the thing still failed. This was unfortunately overlaid by a cron job for repairing the now more often seen root owned directories. The cron job erroneously modified the top layer directories to belong to root:bin, thereby failing all our transfers. In the afternoon the system seemed to have recovered as far as we could tell, but we still do not understand all the reasons for the different errors. Decided to migrate to chimera the next day, if the system proofs to be stable in the next night (so that at least we can assume that the DB is not hopelessly corrupted). ---+ 2. 09. 2009 Again, Good transfer quality <pre> generated on Wed Sep 2 08:45:01 CEST 2009 ------------------------ given starttime 2009-09-01 18:45:03 given endtime 2009-09-02 06:45:03 ============== ERROR ANALYSIS ============== Data base Errors ================== Expired tasks ================== Total: 0 Error message statistics per site: =================================== *** ERRORS from T1_TW_ASGC_Buffer:*** 1 DESTINATION error during PREPARATION phase: [CONNECTION] [srm2__srmPrepareToPut] failed: SOAP-ENV:Client - CGSI-gSOAP: Could not open connection ! 1 SOURCE error during PREPARATION phase: [REQUEST_TIMEOUT] failed to prepare source file in 300 seconds *** ERRORS from T1_US_FNAL_Buffer:*** 96 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full) 27 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : directory creation failed: /pnfs/lcg.cscs.ch/cms/trivcat/store/mc/Summer09/ppMuX/GEN-SIM-RECO/MC_31X_V3-v1/0004 reason: File exists 2 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path 2 SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] FAILED: at [date] state Failed : ThePinCallbacks Timeout 1 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full) 1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Stream ended before EOD/Data channel from /131.225.189.76:33581 was closed before EOD marker) 1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Transfer failed: Stream ended before EOD/Data channel from /131.225.189.6:57905 was closed before EOD marker) 1 SOURCE error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] FAILED: at [date] state Failed : No Route to cell for packet {uoid=stuff} 1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out SITE STATISTICS: ================== first entry: 2009-09-01 18:45:06 last entry: 2009-09-02 06:45:00 T1_CH_CERN_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T1_DE_FZK_Buffer (OK: 1 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 2.5 GB ( 0.1 MB/s) T1_ES_PIC_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T1_FR_CCIN2P3_Buffer (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 24.7 GB ( 0.6 MB/s) T1_IT_CNAF_Buffer (OK: 88 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 101.8 GB ( 2.4 MB/s) T1_TW_ASGC_Buffer (OK: 12 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 85.7 % total: 24.6 GB ( 0.6 MB/s) T1_UK_RAL_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 22.2 GB ( 0.5 MB/s) T1_US_FNAL_Buffer (OK: 2211 Err: 132 Exp: 0 Canc: 0 Lost: 0) succ.: 94.4 % total: 3486.0 GB (80.7 MB/s) T2_DE_RWTH (OK: 180 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 435.4 GB (10.1 MB/s) TOTAL SUMMARY: ================== first entry: 2009-09-01 18:45:06 last entry: 2009-09-02 06:45:00 total transferred: 3856.0 GB in 12.0 hours avg. total rate: 91.4 MB/s = 731.3 Mb/s = 7713.0 GB/day </pre> -- Main.DerekFeichtinger - 2009-08-31 ---------------- %ICON{arrowleft}% Go to [[CMSSiteLog15][previous page]] / [[CMSSiteLog17][next page]] of CMS site log %M%
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r4
<
r3
<
r2
<
r1
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r4 - 2009-09-17
-
DerekFeichtinger
LCGTier2
Log In
(Topic)
LCGTier2 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Users
Entry point / Contact
RoadMap
ATLAS Pages
CMS Pages
CMS User Howto
CHIPP CB
Outreach
Technical
Cluster details
Services
Hardware and OS
Tools & Tips
Monitoring
Logs
Maintenances
Meetings
Tests
Issues
Blog
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
LCGTier2 Web
Users
Groups
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Warning: Can't find topic "".""
Account
Log In
E
dit
A
ttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback