Tags:
tag this topic
create new tag
view all tags
<!-- keep this as a security measure: * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup #uncomment this if you want the page only be viewable by the internal people # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup --> %TOC% %ICON{arrowleft}% Go to [[CMSSiteLogXX][previous page]] / [[CMSSiteLogXX][next page]] of CMS site log %M% ---+ 16. 04. 2010 Full T2-T2 interconnectivity testing ---++ downloads to T2_CH_CSCS All error details of sites with only errors are marked in %BLUE%blue%ENDCOLOR% <pre> generated on Fri Apr 16 09:30:01 CEST 2010 ------------------------ given starttime 2010-04-15 19:30:02 given endtime 2010-04-16 07:30:02 ============== ERROR ANALYSIS ============== Data base Errors ================== Expired tasks ================== Total: 4 Error message statistics per site: =================================== *** ERRORS from T1_DE_KIT_Buffer:*** 7 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out %BLUE% *** ERRORS from T2_UA_KIPT:*** 6 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 530 Login incorrect. : No local mapping 4 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path %ENDCOLOR% *** ERRORS from T2_US_MIT:*** 1 (null) %BLUE% *** ERRORS from T2_US_Florida:*** 15 SOURCE error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://srmb.ihepa.ufl.edu:8443/srm/v2/server]. Givin' up after 3 tries %ENDCOLOR% *** ERRORS from T2_US_Wisconsin:*** 2 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://cmssrm.hep.wisc.edu:8443/srm/managerv2]. Givin' up after 3 tries *** ERRORS from T2_US_UCSD:*** 1 (null) %BLUE% *** ERRORS from T2_PL_Warsaw:*** 4 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist 3 transfer expired in the PhEDEx download agent queue after [hours] h %ENDCOLOR% *** ERRORS from T2_FR_IPHC:*** 2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out *** ERRORS from T2_TW_Taiwan:*** 1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out *** ERRORS from T2_US_Caltech:*** 1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full) 1 SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out %BLUE% *** ERRORS from T2_IT_Bari:*** 13 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries %ENDCOLOR% %BLUE% *** ERRORS from T2_RU_PNPI:*** 4 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out 1 transfer expired in the PhEDEx download agent queue after [hours] h %ENDCOLOR% SITE STATISTICS: ================== first entry: 2010-04-15 19:30:13 last entry: 2010-04-16 07:29:49 T1_CH_CERN_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T1_DE_KIT_Buffer (OK: 200 Err: 7 Exp: 0 Canc: 0 Lost: 0) succ.: 96.6 % total: 930.2 GB (21.5 MB/s) T1_ES_PIC_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 37.6 GB ( 0.9 MB/s) T1_FR_CCIN2P3_Buffer (OK: 18 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 44.4 GB ( 1.0 MB/s) T1_IT_CNAF_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 37.5 GB ( 0.9 MB/s) T1_TW_ASGC_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 39.3 GB ( 0.9 MB/s) T1_UK_RAL_Buffer (OK: 16 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 44.4 GB ( 1.0 MB/s) T1_US_FNAL_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 38.4 GB ( 0.9 MB/s) T2_AT_Vienna (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_BE_IIHE (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_BE_UCL (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 20.1 GB ( 0.5 MB/s) T2_BR_SPRACE (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_BR_UERJ (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.7 GB ( 0.4 MB/s) T2_CN_Beijing (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_DE_DESY (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 15.7 GB ( 0.4 MB/s) T2_DE_RWTH (OK: 2 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 4.8 GB ( 0.1 MB/s) T2_EE_Estonia (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_ES_CIEMAT (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 17.2 GB ( 0.4 MB/s) T2_FI_HIP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_FR_CCIN2P3 (OK: 18 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 44.4 GB ( 1.0 MB/s) T2_FR_GRIF_IRFU (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 24.7 GB ( 0.6 MB/s) T2_FR_GRIF_LLR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 19.7 GB ( 0.5 MB/s) T2_FR_IPHC (OK: 7 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 77.8 % total: 18.8 GB ( 0.4 MB/s) T2_HU_Budapest (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 24.7 GB ( 0.6 MB/s) T2_IN_TIFR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_IT_Bari (OK: 0 Err: 13 Exp: 0 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_IT_Legnaro (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 25.5 GB ( 0.6 MB/s) T2_IT_Pisa (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 17.2 GB ( 0.4 MB/s) T2_IT_Rome (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 26.8 GB ( 0.6 MB/s) T2_KR_KNU (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_PL_Warsaw (OK: 0 Err: 4 Exp: 3 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_PT_LIP_Lisbon (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_PT_NCG_Lisbon (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_RU_IHEP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_RU_INR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_RU_ITEP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_RU_PNPI (OK: 0 Err: 4 Exp: 1 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_RU_RRC_KI (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 26.8 GB ( 0.6 MB/s) T2_RU_SINP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_TR_METU (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_TW_Taiwan (OK: 8 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 88.9 % total: 22.4 GB ( 0.5 MB/s) T2_UA_KIPT (OK: 0 Err: 10 Exp: 0 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_UK_London_Brunel (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 26.8 GB ( 0.6 MB/s) T2_UK_London_IC (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_UK_SGrid_Bristol (OK: 18 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 10.1 GB ( 0.2 MB/s) T2_UK_SGrid_RALPP (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 27.5 GB ( 0.6 MB/s) T2_US_Caltech (OK: 158 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 98.8 % total: 433.7 GB (10.0 MB/s) T2_US_Florida (OK: 0 Err: 15 Exp: 0 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_US_MIT (OK: 8 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 88.9 % total: 22.0 GB ( 0.5 MB/s) T2_US_Nebraska (OK: 120 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 339.6 GB ( 7.9 MB/s) T2_US_Purdue (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s) T2_US_UCSD (OK: 146 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 99.3 % total: 391.9 GB ( 9.1 MB/s) T2_US_Wisconsin (OK: 8 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 80.0 % total: 21.9 GB ( 0.5 MB/s) TOTAL SUMMARY: ================== first entry: 2010-04-15 19:30:13 last entry: 2010-04-16 07:29:49 total transferred: 2935.9 GB in 12.0 hours avg. total rate: 69.6 MB/s = 557.0 Mb/s = 5875.0 GB/day </pre> ---++ Investigating some of the systematic errors ---+++ T2_PL_Warsaw: [USER_ERROR] source file doesn't exist (also seems a problem at Bari and Wisconsin source sites) There's a number of source sites exhibiting this problem <pre> phedex@t3ui01 Utilities]$ ./ErrorQuery --db /home/phedex/config/DBParam.PSI:Debug/PSI -c -s "-12 hours" -e "%USER_ERROR%" -x -m 500 2010-04-16 09:04:17: ErrorQuery[22514]: (re)connecting to database 2010-04-16 09:04:34: ErrorQuery[22514]: disconnected from database #Number of results: 68 (of max 500. Primary search retrieved 68) # #count src dst backend stech dtech fts channel nfiles 7 T2_IT_Bari T1_FR_CCIN2P3_Buffer FTS cms pnfs cclcgftsprod.in2p3.fr STAR-IN2P3 1,2,3 5 T2_IT_Bari T2_DE_DESY FTS cms pnfs fts-fzk.gridka.de STAR-DESY 1,3 6 T2_IT_Bari T1_US_FNAL_Buffer FTS cms 11 cmsfts1.fnal.gov STAR-FNAL 1,2 7 T2_IT_Bari T1_DE_KIT_Buffer FTS cms pnfs fts-fzk.gridka.de STAR-FZK 1,2 3 T2_PL_Warsaw T2_CH_CSCS FTS dpm pnfs fts-fzk.gridka.de STAR-CSCS 3 9 T2_PL_Warsaw T2_IT_Pisa FTS dpm pnfs fts.cr.cnaf.infn.it <None>,STAR-PISA 1,2,3 1 T2_US_Wisconsin T1_ES_PIC_Buffer FTS pnfs pnfs fts.pic.es STAR-PIC 1 3 T2_US_Wisconsin T1_US_FNAL_Buffer FTS pnfs 11 cmsfts1.fnal.gov WISCONSIN-FNAL 1,2 10 T2_US_Wisconsin T1_CH_CERN_Buffer FTS pnfs castor fts-t2-service.cern.ch STAR-CERN 2 3 T2_US_Wisconsin T2_DE_DESY FTS pnfs pnfs fts-fzk.gridka.de STAR-DESY 1,2 2 T2_US_Wisconsin T2_BR_UERJ FTS pnfs pnfs cmsfts1.fnal.gov STAR-UERJ 1 1 T2_US_Wisconsin T1_FR_CCIN2P3_Buffer FTS pnfs pnfs cclcgftsprod.in2p3.fr STAR-IN2P3 3 4 T2_US_Wisconsin T1_TW_ASGC_Buffer FTS pnfs castor w-fts.grid.sinica.edu.tw STAR-ASGC 1,2,3 6 T2_US_Wisconsin T1_DE_KIT_Buffer FTS pnfs pnfs fts-fzk.gridka.de STAR-FZK 1,2,5 1 T2_US_Wisconsin T1_IT_CNAF_Buffer FTS pnfs cms fts.cr.cnaf.infn.it STAR-CNAF 3 #Error Classification: 68 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist </pre> ---+++ T2_IT_Bari: [HTTP_TIMEOUT] failed to contact on remote SRM Looking at the errors in the DB for Bari as source site, there are multiple destinations showing this error. <pre> ./ErrorSiteQuery --db /home/phedex/config/DBParam.PSI:Debug/PSI --src "%Bari%" -m 500 -s "-12 hours" 2010-04-16 09:08:43: ErrorSiteQuery[22535]: (re)connecting to database 2010-04-16 09:08:50: ErrorSiteQuery[22535]: disconnected from database Results starting from date 1271365723 Thu Apr 15 23:08:43 2010 Number of results: 153 (of max 500) **** from T2_IT_Bari to T1_DE_KIT_Buffer: 7 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist 7 TRANSFER error during TRANSFER phase: [TRANSFER_MARKERS_TIMEOUT] No transfer markers received for more than 300 seconds 2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out **** from T2_IT_Bari to T2_AT_Vienna: 1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out **** from T2_IT_Bari to T2_DE_DESY: 5 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist **** from T2_IT_Bari to T2_IT_Pisa: 33 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries **** from T2_IT_Bari to T1_US_FNAL_Buffer: 6 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist **** from T2_IT_Bari to T2_US_Wisconsin: 3 DESTINATION error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://cmssrm.hep.wisc.edu:8443/srm/managerv2]. Givin' up after 3 tries 1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Not in trash: 00060000000000000CB12580) **** from T2_IT_Bari to T2_ES_IFCA: 14 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries **** from T2_IT_Bari to T2_IT_Legnaro: 2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out **** from T2_IT_Bari to T1_IT_CNAF_Buffer: 1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out **** from T2_IT_Bari to T1_CH_CERN_Buffer: 2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out **** from T2_IT_Bari to T1_FR_CCIN2P3_Buffer: 7 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist 1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 500 500-Command failed. : callback failed. 500-globus_xio: System error in writev: Connectio...[error cut] **** from T2_IT_Bari to T2_US_Nebraska: 1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out **** from T2_IT_Bari to T2_ES_CIEMAT: 23 [undefined error message] **** from T2_IT_Bari to T2_US_Purdue: 12 AGENT error during ALLOCATION phase: [CONFIGURATION_ERROR] No Channel found, Channel closed for your VO or VO not authorized for transferring between INFN-BARI and PURDUE-STEELE 6 bypassing transfer due to agent restart 3 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out 1 (null) **** from T2_IT_Bari to T2_CH_CSCS: 15 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries </pre> ---+ 19. 04. 2010 Full T2-T2 testing. High throughput of about 10 TB/day <pre> generated on Mon Apr 19 10:15:01 CEST 2010 ------------------------ given starttime 2010-04-18 20:15:03 given endtime 2010-04-19 08:15:03 ============== ERROR ANALYSIS ============== Data base Errors ================== 3 ORA-02291: integrity constraint (CMS_TRANSFERMGMT_SC.FK_XFER_TASK_DON 1 2010-04-18 23:35:07 0 2010-04-18 23:38:30 0 2010-04-18 23:41:53 0 2010-04-18 23:45:15 1 2010-04-18 23:48:38 0 2010-04-18 23:52:01 0 2010-04-18 23:55:24 0 2010-04-18 23:58:47 0 2010-04-19 00:02:09 1 2010-04-19 00:05:32 Expired tasks ================== Total: 534 Error message statistics per site: =================================== *** ERRORS from T2_UA_KIPT:*** 4 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path 3 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 530 Login incorrect. : No local mapping 2 transfer expired in the PhEDEx download agent queue after [hours] h *** ERRORS from T2_IT_Pisa:*** 1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out *** ERRORS from T2_US_MIT:*** 163 transfer expired in the PhEDEx download agent queue after [hours] h 20 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://se01.cmsaf.mit.edu:8443/srm/managerv2]. Givin' up after 3 tries 2 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_xio: Unable to connect to HIBAT0001.CMSAF.MIT.EDU:2811 globus_xio: globus_libc_getaddrinfo failed. globus_common: Name or service not known 1 SOURCE error during TRANSFER_PREPARATION phase: [LOCALITY] Source file [[srm-URL] locality is UNAVAILABLE 1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 451 Internal timeout 1 SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out *** ERRORS from T1_US_FNAL_Buffer:*** 366 transfer expired in the PhEDEx download agent queue after [hours] h *** ERRORS from T2_US_Florida:*** 10 SOURCE error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://srmb.ihepa.ufl.edu:8443/srm/v2/server]. Givin' up after 3 tries 1 transfer expired in the PhEDEx download agent queue after [hours] h *** ERRORS from T2_US_Wisconsin:*** 1 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path 1 SOURCE error during TRANSFER phase: [SECURITY_ERROR] globus_ftp_client: the server responded with an error 535 Authentication failed: GSSException: Defective credential detected [Caused by: [Caused by: Bad sequence size: 2]] *** ERRORS from T2_RU_PNPI:*** 10 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out 1 transfer expired in the PhEDEx download agent queue after [hours] h *** ERRORS from T1_UK_RAL_Buffer:*** 1 transfer expired in the PhEDEx download agent queue after [hours] h SITE STATISTICS: ================== first entry: 2010-04-18 22:05:27 last entry: 2010-04-19 08:14:59 T1_CH_CERN_Buffer (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T1_DE_KIT_Buffer (OK: 632 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 737.1 GB (20.2 MB/s) T1_ES_PIC_Buffer (OK: 12 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 32.3 GB ( 0.9 MB/s) T1_FR_CCIN2P3_Buffer (OK: 227 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 361.2 GB ( 9.9 MB/s) T1_IT_CNAF_Buffer (OK: 413 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 495.8 GB (13.6 MB/s) T1_TW_ASGC_Buffer (OK: 12 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 33.7 GB ( 0.9 MB/s) T1_UK_RAL_Buffer (OK: 352 Err: 0 Exp: 1 Canc: 0 Lost: 0) succ.: 100.0 % total: 897.8 GB (24.5 MB/s) T1_US_FNAL_Buffer (OK: 86 Err: 0 Exp: 366 Canc: 0 Lost: 0) succ.: 100.0 % total: 186.3 GB ( 5.1 MB/s) T2_AT_Vienna (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_BE_IIHE (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_BE_UCL (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 15.1 GB ( 0.4 MB/s) T2_BR_SPRACE (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_BR_UERJ (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.3 GB ( 0.6 MB/s) T2_CN_Beijing (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_DE_DESY (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 15.7 GB ( 0.4 MB/s) T2_DE_RWTH (OK: 7 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 33.9 GB ( 0.9 MB/s) T2_EE_Estonia (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_ES_CIEMAT (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 17.1 GB ( 0.5 MB/s) T2_ES_IFCA (OK: 4 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 11.3 GB ( 0.3 MB/s) T2_FI_HIP (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_FR_CCIN2P3 (OK: 12 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 29.6 GB ( 0.8 MB/s) T2_FR_GRIF_IRFU (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 19.8 GB ( 0.5 MB/s) T2_FR_GRIF_LLR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 19.7 GB ( 0.5 MB/s) T2_FR_IPHC (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_HU_Budapest (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 14.8 GB ( 0.4 MB/s) T2_IN_TIFR (OK: 4 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 10.7 GB ( 0.3 MB/s) T2_IT_Legnaro (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 20.5 GB ( 0.6 MB/s) T2_IT_Pisa (OK: 6 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 85.7 % total: 17.2 GB ( 0.5 MB/s) T2_IT_Rome (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.6 MB/s) T2_KR_KNU (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_PT_LIP_Lisbon (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_PT_NCG_Lisbon (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.6 MB/s) T2_RU_IHEP (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_RU_INR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.6 MB/s) T2_RU_PNPI (OK: 0 Err: 10 Exp: 1 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_RU_RRC_KI (OK: 4 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 10.7 GB ( 0.3 MB/s) T2_RU_SINP (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_TR_METU (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.6 MB/s) T2_TW_Taiwan (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 22.4 GB ( 0.6 MB/s) T2_UA_KIPT (OK: 0 Err: 7 Exp: 2 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_UK_London_Brunel (OK: 4 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 10.7 GB ( 0.3 MB/s) T2_UK_London_IC (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s) T2_UK_SGrid_Bristol (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 8.3 GB ( 0.2 MB/s) T2_UK_SGrid_RALPP (OK: 4 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 11.0 GB ( 0.3 MB/s) T2_US_Caltech (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 22.0 GB ( 0.6 MB/s) T2_US_Florida (OK: 0 Err: 10 Exp: 1 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s) T2_US_MIT (OK: 312 Err: 25 Exp: 163 Canc: 0 Lost: 0) succ.: 92.6 % total: 662.9 GB (18.1 MB/s) T2_US_Nebraska (OK: 311 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 637.9 GB (17.4 MB/s) T2_US_UCSD (OK: 139 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 189.7 GB ( 5.2 MB/s) T2_US_Wisconsin (OK: 5 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 71.4 % total: 13.7 GB ( 0.4 MB/s) T3_CH_PSI (OK: 1 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 0.0 GB ( 0.0 MB/s) TOTAL SUMMARY: ================== first entry: 2010-04-18 22:05:27 last entry: 2010-04-19 08:14:59 total transferred: 4540.6 GB in 10.2 hours avg. total rate: 127.1 MB/s = 1017.1 Mb/s = 10727.1 GB/day </pre> -- Main.DerekFeichtinger - 2010-04-16 ---------------- %ICON{arrowleft}% Go to [[CMSSiteLogXX][previous page]] / [[CMSSiteLogXX][next page]] of CMS site log %M%
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r4
<
r3
<
r2
<
r1
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r4 - 2010-08-03
-
DerekFeichtinger
LCGTier2
Log In
(Topic)
LCGTier2 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Users
Entry point / Contact
RoadMap
ATLAS Pages
CMS Pages
CMS User Howto
CHIPP CB
Outreach
Technical
Cluster details
Services
Hardware and OS
Tools & Tips
Monitoring
Logs
Maintenances
Meetings
Tests
Issues
Blog
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
LCGTier2 Web
Users
Groups
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Warning: Can't find topic "".""
Account
Log In
E
dit
A
ttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback