Go to
previous page /
next page of CMS site log
16. 04. 2010 Full T2-T2 interconnectivity testing
downloads to T2_CH_CSCS
All error details of sites with only errors are marked in
blue
generated on Fri Apr 16 09:30:01 CEST 2010
------------------------
given starttime 2010-04-15 19:30:02
given endtime 2010-04-16 07:30:02
==============
ERROR ANALYSIS
==============
Data base Errors
==================
Expired tasks
==================
Total: 4
Error message statistics per site:
===================================
*** ERRORS from T1_DE_KIT_Buffer:***
7 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
*** ERRORS from T2_UA_KIPT:***
6 SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 530 Login incorrect. : No local mapping
4 DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path
*** ERRORS from T2_US_MIT:***
1 (null)
*** ERRORS from T2_US_Florida:***
15 SOURCE error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://srmb.ihepa.ufl.edu:8443/srm/v2/server]. Givin' up after 3 tries
*** ERRORS from T2_US_Wisconsin:***
2 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://cmssrm.hep.wisc.edu:8443/srm/managerv2]. Givin' up after 3 tries
*** ERRORS from T2_US_UCSD:***
1 (null)
*** ERRORS from T2_PL_Warsaw:***
4 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
3 transfer expired in the PhEDEx download agent queue after [hours] h
*** ERRORS from T2_FR_IPHC:***
2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
*** ERRORS from T2_TW_Taiwan:***
1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
*** ERRORS from T2_US_Caltech:***
1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
1 SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out
*** ERRORS from T2_IT_Bari:***
13 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries
*** ERRORS from T2_RU_PNPI:***
4 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
1 transfer expired in the PhEDEx download agent queue after [hours] h
SITE STATISTICS:
==================
first entry: 2010-04-15 19:30:13 last entry: 2010-04-16 07:29:49
T1_CH_CERN_Buffer (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T1_DE_KIT_Buffer (OK: 200 Err: 7 Exp: 0 Canc: 0 Lost: 0) succ.: 96.6 % total: 930.2 GB (21.5 MB/s)
T1_ES_PIC_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 37.6 GB ( 0.9 MB/s)
T1_FR_CCIN2P3_Buffer (OK: 18 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 44.4 GB ( 1.0 MB/s)
T1_IT_CNAF_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 37.5 GB ( 0.9 MB/s)
T1_TW_ASGC_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 39.3 GB ( 0.9 MB/s)
T1_UK_RAL_Buffer (OK: 16 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 44.4 GB ( 1.0 MB/s)
T1_US_FNAL_Buffer (OK: 14 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 38.4 GB ( 0.9 MB/s)
T2_AT_Vienna (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_BE_IIHE (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_BE_UCL (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 20.1 GB ( 0.5 MB/s)
T2_BR_SPRACE (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_BR_UERJ (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.7 GB ( 0.4 MB/s)
T2_CN_Beijing (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_DE_DESY (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 15.7 GB ( 0.4 MB/s)
T2_DE_RWTH (OK: 2 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 4.8 GB ( 0.1 MB/s)
T2_EE_Estonia (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 16.1 GB ( 0.4 MB/s)
T2_ES_CIEMAT (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 17.2 GB ( 0.4 MB/s)
T2_FI_HIP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_FR_CCIN2P3 (OK: 18 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 44.4 GB ( 1.0 MB/s)
T2_FR_GRIF_IRFU (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 24.7 GB ( 0.6 MB/s)
T2_FR_GRIF_LLR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 19.7 GB ( 0.5 MB/s)
T2_FR_IPHC (OK: 7 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 77.8 % total: 18.8 GB ( 0.4 MB/s)
T2_HU_Budapest (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 24.7 GB ( 0.6 MB/s)
T2_IN_TIFR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_IT_Bari (OK: 0 Err: 13 Exp: 0 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s)
T2_IT_Legnaro (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 25.5 GB ( 0.6 MB/s)
T2_IT_Pisa (OK: 6 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 17.2 GB ( 0.4 MB/s)
T2_IT_Rome (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 26.8 GB ( 0.6 MB/s)
T2_KR_KNU (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_PL_Warsaw (OK: 0 Err: 4 Exp: 3 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s)
T2_PT_LIP_Lisbon (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_PT_NCG_Lisbon (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_RU_IHEP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_RU_INR (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_RU_ITEP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_RU_PNPI (OK: 0 Err: 4 Exp: 1 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s)
T2_RU_RRC_KI (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 26.8 GB ( 0.6 MB/s)
T2_RU_SINP (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_TR_METU (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_TW_Taiwan (OK: 8 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 88.9 % total: 22.4 GB ( 0.5 MB/s)
T2_UA_KIPT (OK: 0 Err: 10 Exp: 0 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s)
T2_UK_London_Brunel (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 26.8 GB ( 0.6 MB/s)
T2_UK_London_IC (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_UK_SGrid_Bristol (OK: 18 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 10.1 GB ( 0.2 MB/s)
T2_UK_SGrid_RALPP (OK: 10 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 27.5 GB ( 0.6 MB/s)
T2_US_Caltech (OK: 158 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 98.8 % total: 433.7 GB (10.0 MB/s)
T2_US_Florida (OK: 0 Err: 15 Exp: 0 Canc: 0 Lost: 0) succ.: 0.0 % total: 0.0 GB ( 0.0 MB/s)
T2_US_MIT (OK: 8 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 88.9 % total: 22.0 GB ( 0.5 MB/s)
T2_US_Nebraska (OK: 120 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 339.6 GB ( 7.9 MB/s)
T2_US_Purdue (OK: 8 Err: 0 Exp: 0 Canc: 0 Lost: 0) succ.: 100.0 % total: 21.5 GB ( 0.5 MB/s)
T2_US_UCSD (OK: 146 Err: 1 Exp: 0 Canc: 0 Lost: 0) succ.: 99.3 % total: 391.9 GB ( 9.1 MB/s)
T2_US_Wisconsin (OK: 8 Err: 2 Exp: 0 Canc: 0 Lost: 0) succ.: 80.0 % total: 21.9 GB ( 0.5 MB/s)
TOTAL SUMMARY:
==================
first entry: 2010-04-15 19:30:13 last entry: 2010-04-16 07:29:49
total transferred: 2935.9 GB in 12.0 hours
avg. total rate: 69.6 MB/s = 557.0 Mb/s = 5875.0 GB/day
Investigating some of the systematic errors
T2_PL_Warsaw: [USER_ERROR] source file doesn't exist (also seems a problem at Bari and Wisconsin source sites)
There's a number of source sites exhibiting this problem
phedex@t3ui01 Utilities]$ ./ErrorQuery --db /home/phedex/config/DBParam.PSI:Debug/PSI -c -s "-12 hours" -e "%USER_ERROR%" -x -m 500
2010-04-16 09:04:17: ErrorQuery[22514]: (re)connecting to database
2010-04-16 09:04:34: ErrorQuery[22514]: disconnected from database
#Number of results: 68 (of max 500. Primary search retrieved 68)
#
#count src dst backend stech dtech fts channel nfiles
7 T2_IT_Bari T1_FR_CCIN2P3_Buffer FTS cms pnfs cclcgftsprod.in2p3.fr STAR-IN2P3 1,2,3
5 T2_IT_Bari T2_DE_DESY FTS cms pnfs fts-fzk.gridka.de STAR-DESY 1,3
6 T2_IT_Bari T1_US_FNAL_Buffer FTS cms 11 cmsfts1.fnal.gov STAR-FNAL 1,2
7 T2_IT_Bari T1_DE_KIT_Buffer FTS cms pnfs fts-fzk.gridka.de STAR-FZK 1,2
3 T2_PL_Warsaw T2_CH_CSCS FTS dpm pnfs fts-fzk.gridka.de STAR-CSCS 3
9 T2_PL_Warsaw T2_IT_Pisa FTS dpm pnfs fts.cr.cnaf.infn.it ,STAR-PISA 1,2,3
1 T2_US_Wisconsin T1_ES_PIC_Buffer FTS pnfs pnfs fts.pic.es STAR-PIC 1
3 T2_US_Wisconsin T1_US_FNAL_Buffer FTS pnfs 11 cmsfts1.fnal.gov WISCONSIN-FNAL 1,2
10 T2_US_Wisconsin T1_CH_CERN_Buffer FTS pnfs castor fts-t2-service.cern.ch STAR-CERN 2
3 T2_US_Wisconsin T2_DE_DESY FTS pnfs pnfs fts-fzk.gridka.de STAR-DESY 1,2
2 T2_US_Wisconsin T2_BR_UERJ FTS pnfs pnfs cmsfts1.fnal.gov STAR-UERJ 1
1 T2_US_Wisconsin T1_FR_CCIN2P3_Buffer FTS pnfs pnfs cclcgftsprod.in2p3.fr STAR-IN2P3 3
4 T2_US_Wisconsin T1_TW_ASGC_Buffer FTS pnfs castor w-fts.grid.sinica.edu.tw STAR-ASGC 1,2,3
6 T2_US_Wisconsin T1_DE_KIT_Buffer FTS pnfs pnfs fts-fzk.gridka.de STAR-FZK 1,2,5
1 T2_US_Wisconsin T1_IT_CNAF_Buffer FTS pnfs cms fts.cr.cnaf.infn.it STAR-CNAF 3
#Error Classification:
68 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
T2_IT_Bari: [HTTP_TIMEOUT] failed to contact on remote SRM
Looking at the errors in the DB for Bari as source site, there are multiple destinations showing this error.
./ErrorSiteQuery --db /home/phedex/config/DBParam.PSI:Debug/PSI --src "%Bari%" -m 500 -s "-12 hours"
2010-04-16 09:08:43: ErrorSiteQuery[22535]: (re)connecting to database
2010-04-16 09:08:50: ErrorSiteQuery[22535]: disconnected from database
Results starting from date 1271365723 Thu Apr 15 23:08:43 2010
Number of results: 153 (of max 500)
**** from T2_IT_Bari to T1_DE_KIT_Buffer:
7 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
7 TRANSFER error during TRANSFER phase: [TRANSFER_MARKERS_TIMEOUT] No transfer markers received for more than 300 seconds
2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
**** from T2_IT_Bari to T2_AT_Vienna:
1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
**** from T2_IT_Bari to T2_DE_DESY:
5 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
**** from T2_IT_Bari to T2_IT_Pisa:
33 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries
**** from T2_IT_Bari to T1_US_FNAL_Buffer:
6 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
**** from T2_IT_Bari to T2_US_Wisconsin:
3 DESTINATION error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://cmssrm.hep.wisc.edu:8443/srm/managerv2]. Givin' up after 3 tries
1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Not in trash: 00060000000000000CB12580)
**** from T2_IT_Bari to T2_ES_IFCA:
14 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries
**** from T2_IT_Bari to T2_IT_Legnaro:
2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
**** from T2_IT_Bari to T1_IT_CNAF_Buffer:
1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
**** from T2_IT_Bari to T1_CH_CERN_Buffer:
2 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
**** from T2_IT_Bari to T1_FR_CCIN2P3_Buffer:
7 SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
1 TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 500 500-Command failed. : callback failed. 500-globus_xio: System error in writev: Connectio...[error cut]
**** from T2_IT_Bari to T2_US_Nebraska:
1 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
**** from T2_IT_Bari to T2_ES_CIEMAT:
23 [undefined error message]
**** from T2_IT_Bari to T2_US_Purdue:
12 AGENT error during ALLOCATION phase: [CONFIGURATION_ERROR] No Channel found, Channel closed for your VO or VO not authorized for transferring between INFN-BARI and PURDUE-STEELE
6 bypassing transfer due to agent restart
3 TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
1 (null)
**** from T2_IT_Bari to T2_CH_CSCS:
15 SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries
--
DerekFeichtinger - 2010-04-16
Go to
previous page /
next page of CMS site log