Go to previous page / next page of CMS site log

6. 6. 2008 Analysis Jobs reading at > 35 MB/s per Job from our Storage

~50 Jobs from Florian Bechtel managed to read with a total of ca. 1600 MB/s from our storage. This is a higher rate as we had seen from any other CMSSW based jobs. Florian reported that he was using CMSSW_2_0_7 running code from QCDAnalysis/UEAnalysis with a configuration from QCDAnalysis/UEAnalysis/test/ueAnalysisRootFileChain.cfg.

Fileservers' and worker's network plots:

dCache default (=dcap) movers plot showing the same curve:

Plots showing distribution of outwards traffic over fileservers:

CE Log analysis

The gatekeeper shows 56 jobs having been accepted for Florian Bechtel as user cms022 between 16:34:25 and 16:40:45. However, the PBS accounting logs show that only 50 were started between 16:35:26 and 16:41:29, so 6 seem to have been rejected:

 PID: 17135 -- Notice: 5: "/C=DE/O=GermanGrid/OU=DESY/CN=Florian Bechtel" mapped to cms022 (18550/1399)
$> grep "user=cms022" /var/spool/pbs/server_priv/accounting/20080606|grep -v "Exit_status"|wc -l
50

Only 49 are recorded with an exit status, and all exited ok with Exit status 0:

$> grep "user=cms022" 20080606|grep "Exit_status"|wc -l
49
$> grep "user=cms022" 20080606|grep "Exit_status"|sed -e 's/.*\(Exit_status=[0-9]*\).*/\1/'|sort|uniq -c
     49 Exit_status=0

Job running time was distributed like this (minutes and seconds cut away by the sed expression):

$> grep "user=cms022" 20080606|grep "Exit_status"|sed -e 's/.*\(resources_used.walltime=[^:]*\).*/\1/'|sort|uniq -c
      4 resources_used.walltime=00
     11 resources_used.walltime=01
     30 resources_used.walltime=02
      4 resources_used.walltime=03

Two jobs almost finished immediately in the first 10 minutes, so we can assume that the load derived from ~47 jobs.

$> grep "user=cms022" 20080606|grep "Exit_status"|sed -ne 's/.*\(resources_used.walltime=00:[^:]*\).*/\1/p'|sort
resources_used.walltime=00:02
resources_used.walltime=00:06
resources_used.walltime=00:16
resources_used.walltime=00:37

Comparison with I/O rate monitoring by CMS dashboard

The dashboard shows that at the peak rate less than 30 jobs were reporting I/O rates. The averaged I/O rates at that time were higher that 40 MB/s. If this is averaged over the reporting jobs (which is not totally clear from the page description), then the dashboard reports similar rates.

Note that there is a 2h time difference between dashboard time and local time.

Further dashboard analysis

Link to the dashboard query. It's not easy to select the correct time range for jobs for which we know the local time span. So, it requires trial and error and no other interfering jobs from that user in a larger time interval.

The jobs ran over different data sets and the RB plot suggests that 4 of the 50 jobs failed due to a RB, since all failures were submitted through it.

Go to previous page / next page of CMS site log

-- DerekFeichtinger - 07 Jun 2008