Tags:
create new tag
view all tags

Arrow left Go to previous page / next page of CMS site log MOVED TO...

16. 04. 2010 Full T2-T2 interconnectivity testing

downloads to T2_CH_CSCS

All error details of sites with only errors are marked in blue

generated on Fri Apr 16 09:30:01 CEST 2010 
------------------------
given starttime 2010-04-15 19:30:02
given endtime 2010-04-16 07:30:02


==============
ERROR ANALYSIS
==============


Data base Errors
==================


Expired tasks
==================
Total: 4


Error message statistics per site:
===================================

 *** ERRORS from T1_DE_KIT_Buffer:***
      7   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out


 *** ERRORS from T2_UA_KIPT:***
      6   SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 530 Login incorrect. : No local mapping
      4   DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path


 *** ERRORS from T2_US_MIT:***
      1   (null)


 *** ERRORS from T2_US_Florida:***
     15   SOURCE error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://srmb.ihepa.ufl.edu:8443/srm/v2/server]. Givin' up after 3 tries


 *** ERRORS from T2_US_Wisconsin:***
      2   SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://cmssrm.hep.wisc.edu:8443/srm/managerv2]. Givin' up after 3 tries

 *** ERRORS from T2_US_UCSD:***
      1   (null)


 *** ERRORS from T2_PL_Warsaw:***
      4   SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
      3   transfer expired in the PhEDEx download agent queue after [hours] h


 *** ERRORS from T2_FR_IPHC:***
      2   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out

 *** ERRORS from T2_TW_Taiwan:***
      1   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out

 *** ERRORS from T2_US_Caltech:***
      1   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] an end-of-file was reached globus_xio: An end of file occurred (possibly the destination disk is full)
      1   SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out

 *** ERRORS from T2_IT_Bari:***
     13   SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries



 *** ERRORS from T2_RU_PNPI:***
      4   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
      1   transfer expired in the PhEDEx download agent queue after [hours] h


SITE STATISTICS:
==================
                         first entry: 2010-04-15 19:30:13      last entry: 2010-04-16 07:29:49
   T1_CH_CERN_Buffer (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
    T1_DE_KIT_Buffer (OK: 200  Err:   7  Exp:   0  Canc:   0  Lost:   0)   succ.: 96.6 %   total:  930.2 GB  (21.5 MB/s)
    T1_ES_PIC_Buffer (OK:  14  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   37.6 GB  ( 0.9 MB/s)
T1_FR_CCIN2P3_Buffer (OK:  18  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   44.4 GB  ( 1.0 MB/s)
   T1_IT_CNAF_Buffer (OK:  14  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   37.5 GB  ( 0.9 MB/s)
   T1_TW_ASGC_Buffer (OK:  14  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   39.3 GB  ( 0.9 MB/s)
    T1_UK_RAL_Buffer (OK:  16  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   44.4 GB  ( 1.0 MB/s)
   T1_US_FNAL_Buffer (OK:  14  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   38.4 GB  ( 0.9 MB/s)
        T2_AT_Vienna (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_BE_IIHE (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
           T2_BE_UCL (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   20.1 GB  ( 0.5 MB/s)
        T2_BR_SPRACE (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_BR_UERJ (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.7 GB  ( 0.4 MB/s)
       T2_CN_Beijing (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_DE_DESY (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   15.7 GB  ( 0.4 MB/s)
          T2_DE_RWTH (OK:   2  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:    4.8 GB  ( 0.1 MB/s)
       T2_EE_Estonia (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
        T2_ES_CIEMAT (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   17.2 GB  ( 0.4 MB/s)
           T2_FI_HIP (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
       T2_FR_CCIN2P3 (OK:  18  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   44.4 GB  ( 1.0 MB/s)
     T2_FR_GRIF_IRFU (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   24.7 GB  ( 0.6 MB/s)
      T2_FR_GRIF_LLR (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   19.7 GB  ( 0.5 MB/s)
          T2_FR_IPHC (OK:   7  Err:   2  Exp:   0  Canc:   0  Lost:   0)   succ.: 77.8 %   total:   18.8 GB  ( 0.4 MB/s)
      T2_HU_Budapest (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   24.7 GB  ( 0.6 MB/s)
          T2_IN_TIFR (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_IT_Bari (OK:   0  Err:  13  Exp:   0  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
       T2_IT_Legnaro (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   25.5 GB  ( 0.6 MB/s)
          T2_IT_Pisa (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   17.2 GB  ( 0.4 MB/s)
          T2_IT_Rome (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   26.8 GB  ( 0.6 MB/s)
           T2_KR_KNU (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
        T2_PL_Warsaw (OK:   0  Err:   4  Exp:   3  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
    T2_PT_LIP_Lisbon (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
    T2_PT_NCG_Lisbon (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_RU_IHEP (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
           T2_RU_INR (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_RU_ITEP (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_RU_PNPI (OK:   0  Err:   4  Exp:   1  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
        T2_RU_RRC_KI (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   26.8 GB  ( 0.6 MB/s)
          T2_RU_SINP (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_TR_METU (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
        T2_TW_Taiwan (OK:   8  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 88.9 %   total:   22.4 GB  ( 0.5 MB/s)
          T2_UA_KIPT (OK:   0  Err:  10  Exp:   0  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
 T2_UK_London_Brunel (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   26.8 GB  ( 0.6 MB/s)
     T2_UK_London_IC (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
 T2_UK_SGrid_Bristol (OK:  18  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   10.1 GB  ( 0.2 MB/s)
   T2_UK_SGrid_RALPP (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   27.5 GB  ( 0.6 MB/s)
       T2_US_Caltech (OK: 158  Err:   2  Exp:   0  Canc:   0  Lost:   0)   succ.: 98.8 %   total:  433.7 GB  (10.0 MB/s)
       T2_US_Florida (OK:   0  Err:  15  Exp:   0  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
           T2_US_MIT (OK:   8  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 88.9 %   total:   22.0 GB  ( 0.5 MB/s)
      T2_US_Nebraska (OK: 120  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  339.6 GB  ( 7.9 MB/s)
        T2_US_Purdue (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.5 MB/s)
          T2_US_UCSD (OK: 146  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 99.3 %   total:  391.9 GB  ( 9.1 MB/s)
     T2_US_Wisconsin (OK:   8  Err:   2  Exp:   0  Canc:   0  Lost:   0)   succ.: 80.0 %   total:   21.9 GB  ( 0.5 MB/s)

TOTAL SUMMARY:
==================
                         first entry: 2010-04-15 19:30:13      last entry: 2010-04-16 07:29:49
total transferred: 2935.9 GB  in 12.0 hours
avg. total rate: 69.6 MB/s = 557.0 Mb/s  = 5875.0 GB/day

Investigating some of the systematic errors

T2_PL_Warsaw: [USER_ERROR] source file doesn't exist (also seems a problem at Bari and Wisconsin source sites)

There's a number of source sites exhibiting this problem

phedex@t3ui01 Utilities]$ ./ErrorQuery --db /home/phedex/config/DBParam.PSI:Debug/PSI -c -s "-12 hours" -e "%USER_ERROR%" -x -m 500
2010-04-16 09:04:17: ErrorQuery[22514]: (re)connecting to database
2010-04-16 09:04:34: ErrorQuery[22514]: disconnected from database
#Number of results: 68 (of max 500. Primary search retrieved 68)
#
#count  src                  dst                  backend    stech  dtech  fts                      channel      nfiles
     7  T2_IT_Bari           T1_FR_CCIN2P3_Buffer FTS        cms    pnfs   cclcgftsprod.in2p3.fr    STAR-IN2P3   1,2,3
     5  T2_IT_Bari           T2_DE_DESY           FTS        cms    pnfs   fts-fzk.gridka.de        STAR-DESY    1,3
     6  T2_IT_Bari           T1_US_FNAL_Buffer    FTS        cms    11     cmsfts1.fnal.gov         STAR-FNAL    1,2
     7  T2_IT_Bari           T1_DE_KIT_Buffer     FTS        cms    pnfs   fts-fzk.gridka.de        STAR-FZK     1,2
     3  T2_PL_Warsaw         T2_CH_CSCS           FTS        dpm    pnfs   fts-fzk.gridka.de        STAR-CSCS      3
     9  T2_PL_Warsaw         T2_IT_Pisa           FTS        dpm    pnfs   fts.cr.cnaf.infn.it      ,STAR-PISA 1,2,3
     1  T2_US_Wisconsin      T1_ES_PIC_Buffer     FTS        pnfs   pnfs   fts.pic.es               STAR-PIC       1
     3  T2_US_Wisconsin      T1_US_FNAL_Buffer    FTS        pnfs   11     cmsfts1.fnal.gov         WISCONSIN-FNAL 1,2
    10  T2_US_Wisconsin      T1_CH_CERN_Buffer    FTS        pnfs   castor fts-t2-service.cern.ch   STAR-CERN      2
     3  T2_US_Wisconsin      T2_DE_DESY           FTS        pnfs   pnfs   fts-fzk.gridka.de        STAR-DESY    1,2
     2  T2_US_Wisconsin      T2_BR_UERJ           FTS        pnfs   pnfs   cmsfts1.fnal.gov         STAR-UERJ      1
     1  T2_US_Wisconsin      T1_FR_CCIN2P3_Buffer FTS        pnfs   pnfs   cclcgftsprod.in2p3.fr    STAR-IN2P3     3
     4  T2_US_Wisconsin      T1_TW_ASGC_Buffer    FTS        pnfs   castor w-fts.grid.sinica.edu.tw STAR-ASGC    1,2,3
     6  T2_US_Wisconsin      T1_DE_KIT_Buffer     FTS        pnfs   pnfs   fts-fzk.gridka.de        STAR-FZK     1,2,5
     1  T2_US_Wisconsin      T1_IT_CNAF_Buffer    FTS        pnfs   cms    fts.cr.cnaf.infn.it      STAR-CNAF      3

#Error Classification:
  68  SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist

T2_IT_Bari: [HTTP_TIMEOUT] failed to contact on remote SRM

Looking at the errors in the DB for Bari as source site, there are multiple destinations showing this error.

./ErrorSiteQuery --db /home/phedex/config/DBParam.PSI:Debug/PSI --src "%Bari%" -m 500 -s "-12 hours"
2010-04-16 09:08:43: ErrorSiteQuery[22535]: (re)connecting to database
2010-04-16 09:08:50: ErrorSiteQuery[22535]: disconnected from database
Results starting from date 1271365723  Thu Apr 15 23:08:43 2010
Number of results: 153 (of max 500)


**** from T2_IT_Bari to T1_DE_KIT_Buffer:
      7   SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
      7   TRANSFER error during TRANSFER phase: [TRANSFER_MARKERS_TIMEOUT] No transfer markers received for more than 300 seconds
      2   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out


**** from T2_IT_Bari to T2_AT_Vienna:
      1   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out


**** from T2_IT_Bari to T2_DE_DESY:
      5   SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist


**** from T2_IT_Bari to T2_IT_Pisa:
     33   SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries


**** from T2_IT_Bari to T1_US_FNAL_Buffer:
      6   SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist


**** from T2_IT_Bari to T2_US_Wisconsin:
      3   DESTINATION error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://cmssrm.hep.wisc.edu:8443/srm/managerv2]. Givin' up after 3 tries
      1   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 426 Transfer aborted (Not in trash: 00060000000000000CB12580)


**** from T2_IT_Bari to T2_ES_IFCA:
     14   SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries


**** from T2_IT_Bari to T2_IT_Legnaro:
      2   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out


**** from T2_IT_Bari to T1_IT_CNAF_Buffer:
      1   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out


**** from T2_IT_Bari to T1_CH_CERN_Buffer:
      2   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out


**** from T2_IT_Bari to T1_FR_CCIN2P3_Buffer:
      7   SOURCE error during TRANSFER_PREPARATION phase: [USER_ERROR] source file doesn't exist
      1   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 500 500-Command failed. : callback failed. 500-globus_xio: System error in writev: Connectio...[error cut]


**** from T2_IT_Bari to T2_US_Nebraska:
      1   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out


**** from T2_IT_Bari to T2_ES_CIEMAT:
     23   [undefined error message]


**** from T2_IT_Bari to T2_US_Purdue:
     12   AGENT error during ALLOCATION phase: [CONFIGURATION_ERROR] No Channel found, Channel closed for your VO or VO not authorized for transferring between INFN-BARI and PURDUE-STEELE
      6   bypassing transfer due to agent restart
      3   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
      1   (null)


**** from T2_IT_Bari to T2_CH_CSCS:
     15   SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://storm-se-01.ba.infn.it:8444/srm/managerv2]. Givin' up after 3 tries

19. 04. 2010 Full T2-T2 testing. High throughput of about 10 TB/day

generated on Mon Apr 19 10:15:01 CEST 2010 
------------------------
given starttime 2010-04-18 20:15:03
given endtime 2010-04-19 08:15:03


==============
ERROR ANALYSIS
==============


Data base Errors
==================
     3  ORA-02291: integrity constraint (CMS_TRANSFERMGMT_SC.FK_XFER_TASK_DON
          1   2010-04-18 23:35:07
          0   2010-04-18 23:38:30
          0   2010-04-18 23:41:53
          0   2010-04-18 23:45:15
          1   2010-04-18 23:48:38
          0   2010-04-18 23:52:01
          0   2010-04-18 23:55:24
          0   2010-04-18 23:58:47
          0   2010-04-19 00:02:09
          1   2010-04-19 00:05:32


Expired tasks
==================
Total: 534


Error message statistics per site:
===================================

 *** ERRORS from T2_UA_KIPT:***
      4   DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path
      3   SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 530 Login incorrect. : No local mapping
      2   transfer expired in the PhEDEx download agent queue after [hours] h

 *** ERRORS from T2_IT_Pisa:***
      1   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out

 *** ERRORS from T2_US_MIT:***
    163   transfer expired in the PhEDEx download agent queue after [hours] h
     20   SOURCE error during TRANSFER_PREPARATION phase: [HTTP_TIMEOUT] failed to contact on remote SRM [httpg://se01.cmsaf.mit.edu:8443/srm/managerv2]. Givin' up after 3 tries
      2   SOURCE error during TRANSFER phase: [GRIDFTP_ERROR] globus_xio: Unable to connect to HIBAT0001.CMSAF.MIT.EDU:2811 globus_xio: globus_libc_getaddrinfo failed. globus_common: Name or service not known
      1   SOURCE error during TRANSFER_PREPARATION phase: [LOCALITY] Source file [[srm-URL] locality is UNAVAILABLE
      1   TRANSFER error during TRANSFER phase: [GRIDFTP_ERROR] globus_ftp_client: the server responded with an error 451 Internal timeout
      1   SOURCE error during TRANSFER phase: [TRANSFER_TIMEOUT] globus_ftp_client_size: Connection timed out

 *** ERRORS from T1_US_FNAL_Buffer:***
    366   transfer expired in the PhEDEx download agent queue after [hours] h

 *** ERRORS from T2_US_Florida:***
     10   SOURCE error during TRANSFER_PREPARATION phase: [CONNECTION_ERROR] failed to contact on remote SRM [httpg://srmb.ihepa.ufl.edu:8443/srm/v2/server]. Givin' up after 3 tries
      1   transfer expired in the PhEDEx download agent queue after [hours] h

 *** ERRORS from T2_US_Wisconsin:***
      1   DESTINATION error during TRANSFER_PREPARATION phase: [GENERAL_FAILURE] at [date] state Failed : Marking Space as Being Used failed =>Already have 1 record(s) with pnfsPath=path
      1   SOURCE error during TRANSFER phase: [SECURITY_ERROR] globus_ftp_client: the server responded with an error 535 Authentication failed: GSSException: Defective credential detected [Caused by: [Caused by: Bad sequence size: 2]]

 *** ERRORS from T2_RU_PNPI:***
     10   TRANSFER error during TRANSFER phase: [TRANSFER_TIMEOUT] gridftp_copy_wait: Connection timed out
      1   transfer expired in the PhEDEx download agent queue after [hours] h

 *** ERRORS from T1_UK_RAL_Buffer:***
      1   transfer expired in the PhEDEx download agent queue after [hours] h

SITE STATISTICS:
==================
                         first entry: 2010-04-18 22:05:27      last entry: 2010-04-19 08:14:59
   T1_CH_CERN_Buffer (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
    T1_DE_KIT_Buffer (OK: 632  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  737.1 GB  (20.2 MB/s)
    T1_ES_PIC_Buffer (OK:  12  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   32.3 GB  ( 0.9 MB/s)
T1_FR_CCIN2P3_Buffer (OK: 227  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  361.2 GB  ( 9.9 MB/s)
   T1_IT_CNAF_Buffer (OK: 413  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  495.8 GB  (13.6 MB/s)
   T1_TW_ASGC_Buffer (OK:  12  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   33.7 GB  ( 0.9 MB/s)
    T1_UK_RAL_Buffer (OK: 352  Err:   0  Exp:   1  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  897.8 GB  (24.5 MB/s)
   T1_US_FNAL_Buffer (OK:  86  Err:   0  Exp: 366  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  186.3 GB  ( 5.1 MB/s)
        T2_AT_Vienna (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
          T2_BE_IIHE (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
           T2_BE_UCL (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   15.1 GB  ( 0.4 MB/s)
        T2_BR_SPRACE (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
          T2_BR_UERJ (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.3 GB  ( 0.6 MB/s)
       T2_CN_Beijing (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
          T2_DE_DESY (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   15.7 GB  ( 0.4 MB/s)
          T2_DE_RWTH (OK:   7  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   33.9 GB  ( 0.9 MB/s)
       T2_EE_Estonia (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
        T2_ES_CIEMAT (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   17.1 GB  ( 0.5 MB/s)
          T2_ES_IFCA (OK:   4  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   11.3 GB  ( 0.3 MB/s)
           T2_FI_HIP (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
       T2_FR_CCIN2P3 (OK:  12  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   29.6 GB  ( 0.8 MB/s)
     T2_FR_GRIF_IRFU (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   19.8 GB  ( 0.5 MB/s)
      T2_FR_GRIF_LLR (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   19.7 GB  ( 0.5 MB/s)
          T2_FR_IPHC (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
      T2_HU_Budapest (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   14.8 GB  ( 0.4 MB/s)
          T2_IN_TIFR (OK:   4  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   10.7 GB  ( 0.3 MB/s)
       T2_IT_Legnaro (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   20.5 GB  ( 0.6 MB/s)
          T2_IT_Pisa (OK:   6  Err:   1  Exp:   0  Canc:   0  Lost:   0)   succ.: 85.7 %   total:   17.2 GB  ( 0.5 MB/s)
          T2_IT_Rome (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.6 MB/s)
           T2_KR_KNU (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
    T2_PT_LIP_Lisbon (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
    T2_PT_NCG_Lisbon (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.6 MB/s)
          T2_RU_IHEP (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
           T2_RU_INR (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.6 MB/s)
          T2_RU_PNPI (OK:   0  Err:  10  Exp:   1  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
        T2_RU_RRC_KI (OK:   4  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   10.7 GB  ( 0.3 MB/s)
          T2_RU_SINP (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
          T2_TR_METU (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   21.5 GB  ( 0.6 MB/s)
        T2_TW_Taiwan (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   22.4 GB  ( 0.6 MB/s)
          T2_UA_KIPT (OK:   0  Err:   7  Exp:   2  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
 T2_UK_London_Brunel (OK:   4  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   10.7 GB  ( 0.3 MB/s)
     T2_UK_London_IC (OK:   6  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   16.1 GB  ( 0.4 MB/s)
 T2_UK_SGrid_Bristol (OK:  10  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:    8.3 GB  ( 0.2 MB/s)
   T2_UK_SGrid_RALPP (OK:   4  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   11.0 GB  ( 0.3 MB/s)
       T2_US_Caltech (OK:   8  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:   22.0 GB  ( 0.6 MB/s)
       T2_US_Florida (OK:   0  Err:  10  Exp:   1  Canc:   0  Lost:   0)   succ.: 0.0 %   total:    0.0 GB  ( 0.0 MB/s)
           T2_US_MIT (OK: 312  Err:  25  Exp: 163  Canc:   0  Lost:   0)   succ.: 92.6 %   total:  662.9 GB  (18.1 MB/s)
      T2_US_Nebraska (OK: 311  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  637.9 GB  (17.4 MB/s)
          T2_US_UCSD (OK: 139  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:  189.7 GB  ( 5.2 MB/s)
     T2_US_Wisconsin (OK:   5  Err:   2  Exp:   0  Canc:   0  Lost:   0)   succ.: 71.4 %   total:   13.7 GB  ( 0.4 MB/s)
           T3_CH_PSI (OK:   1  Err:   0  Exp:   0  Canc:   0  Lost:   0)   succ.: 100.0 %   total:    0.0 GB  ( 0.0 MB/s)

TOTAL SUMMARY:
==================
                         first entry: 2010-04-18 22:05:27      last entry: 2010-04-19 08:14:59
total transferred: 4540.6 GB  in 10.2 hours
avg. total rate: 127.1 MB/s = 1017.1 Mb/s  = 10727.1 GB/day

-- DerekFeichtinger - 2010-04-16


Arrow left Go to previous page / next page of CMS site log MOVED TO...

Edit | Attach | Watch | Print version | History: r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r4 - 2010-08-03 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback