create new tag
view all tags

Arrow left Go to previous page / next page of CMS site log MOVED TO...

01. 02. 2008 Comparison of dashboard monitoring output and CSCS local monitoring.

A first comparison had been done in October (q.v. CMSSiteLog8).

Dashboard plots

The new dashboard contains functionality which should be able to reflect the currently running jobs at a given time, and not only categorize jobs according to their submission time.

Looking at the result of a dashboard query of what ran on ce01.lcg.cscs.ch this morning from 3:20h to 4:20h (link to the query) :


Sorted according to users (link to the query)


Local ganglia monitoring

The local monitoring is done with ganglia/gmetad and therefore the history is kept in the typical round robin databases with different integration slices (accurate for last hour, then gets integrated to the day RRD, etc.), so it's not very exact for looking at the older historic data and at short processes. I wanted to look at job robot jobs here, which are short processes. I turns out that some of the ganglia information is not reliable for these, because the job robot jobs run for about 1 min while the sampling interval for the CE job counting was 5 min. But the log excerpts further below are exact, and the movers and I/O measurements are sampled at 1 min intervals.

Number of jobs on the CE:


The number of active analysis jobs one can get indirectly by looking at the number of active CMS LAN transfers.


For completeness, here are the IO rates on the currently active file servers (stacked), so one can estimate the I/O from that, noting that the CMS spike of short jobs (job robot characteristic), is reflected in the "bytes out" graph.


Values from CE gatekeeper logs

There were 50 jobs from Vincenzo Miccio which mainly seem to be responsible for the peak seen between 3:30h and 4:10h.

02/01/2008 02:23:42 cms016 Exit_status=0 cput=00:03:49 mem=342868kb vmem=554140kb walltime=00:14:49

02/01/2008 03:35:56 cms016 Exit_status=0 cput=00:04:19 mem=351268kb vmem=559636kb walltime=00:05:31
02/01/2008 03:36:59 cms016 Exit_status=0 cput=00:03:45 mem=340740kb vmem=572996kb walltime=00:04:55
02/01/2008 03:37:23 cms016 Exit_status=0 cput=00:04:00 mem=357404kb vmem=572052kb walltime=00:05:10
02/01/2008 03:41:07 cms016 Exit_status=0 cput=00:04:06 mem=380740kb vmem=552072kb walltime=00:05:09
02/01/2008 03:41:58 cms016 Exit_status=0 cput=00:03:45 mem=313328kb vmem=525808kb walltime=00:04:57
02/01/2008 03:42:28 cms016 Exit_status=0 cput=00:03:57 mem=339080kb vmem=550204kb walltime=00:05:03
02/01/2008 03:44:02 cms016 Exit_status=0 cput=00:03:48 mem=351284kb vmem=576344kb walltime=00:05:08
02/01/2008 03:45:39 cms016 Exit_status=0 cput=00:03:57 mem=307612kb vmem=518232kb walltime=00:05:09
02/01/2008 03:45:59 cms016 Exit_status=0 cput=00:03:48 mem=334840kb vmem=539416kb walltime=00:04:50
02/01/2008 03:46:29 cms016 Exit_status=0 cput=00:03:52 mem=319300kb vmem=550716kb walltime=00:05:03
02/01/2008 03:46:52 cms016 Exit_status=0 cput=00:03:41 mem=318588kb vmem=529516kb walltime=00:04:51
02/01/2008 03:47:26 cms016 Exit_status=0 cput=00:03:52 mem=348604kb vmem=564200kb walltime=00:04:56
02/01/2008 03:47:35 cms016 Exit_status=0 cput=00:03:50 mem=323724kb vmem=530560kb walltime=00:05:00
02/01/2008 03:48:56 cms016 Exit_status=0 cput=00:03:35 mem=330376kb vmem=545200kb walltime=00:04:51
02/01/2008 03:49:29 cms016 Exit_status=0 cput=00:04:12 mem=348992kb vmem=547552kb walltime=00:05:54
02/01/2008 03:50:45 cms016 Exit_status=0 cput=00:03:56 mem=407676kb vmem=655292kb walltime=00:05:04
02/01/2008 03:51:27 cms016 Exit_status=0 cput=00:04:14 mem=375472kb vmem=577292kb walltime=00:05:26
02/01/2008 03:51:38 cms016 Exit_status=0 cput=00:03:51 mem=313628kb vmem=528932kb walltime=00:04:54
02/01/2008 03:51:44 cms016 Exit_status=0 cput=00:03:57 mem=334288kb vmem=524532kb walltime=00:05:14
02/01/2008 03:51:51 cms016 Exit_status=0 cput=00:04:04 mem=315504kb vmem=520436kb walltime=00:05:40
02/01/2008 03:52:01 cms016 Exit_status=0 cput=00:04:04 mem=372364kb vmem=570316kb walltime=00:05:55
02/01/2008 03:52:10 cms016 Exit_status=0 cput=00:04:09 mem=314304kb vmem=527580kb walltime=00:05:17
02/01/2008 03:52:43 cms016 Exit_status=0 cput=00:03:49 mem=345092kb vmem=577756kb walltime=00:05:15
02/01/2008 03:52:56 cms016 Exit_status=0 cput=00:04:16 mem=309684kb vmem=518520kb walltime=00:05:19
02/01/2008 03:53:40 cms016 Exit_status=0 cput=00:04:09 mem=316772kb vmem=516836kb walltime=00:05:29
02/01/2008 03:54:01 cms016 Exit_status=0 cput=00:03:49 mem=310920kb vmem=510740kb walltime=00:05:04
02/01/2008 03:55:56 cms016 Exit_status=0 cput=00:03:55 mem=333968kb vmem=552764kb walltime=00:05:24
02/01/2008 03:55:58 cms016 Exit_status=0 cput=00:04:31 mem=345156kb vmem=550768kb walltime=00:06:26
02/01/2008 03:56:03 cms016 Exit_status=0 cput=00:04:08 mem=350256kb vmem=563136kb walltime=00:05:15
02/01/2008 03:56:34 cms016 Exit_status=0 cput=00:04:01 mem=384556kb vmem=586284kb walltime=00:05:05
02/01/2008 03:56:52 cms016 Exit_status=0 cput=00:04:01 mem=335652kb vmem=557328kb walltime=00:05:06
02/01/2008 03:57:08 cms016 Exit_status=0 cput=00:03:49 mem=331444kb vmem=552352kb walltime=00:05:16
02/01/2008 03:57:22 cms016 Exit_status=0 cput=00:03:45 mem=332720kb vmem=527792kb walltime=00:05:08
02/01/2008 03:58:02 cms016 Exit_status=0 cput=00:04:24 mem=336848kb vmem=556476kb walltime=00:06:23
02/01/2008 03:58:04 cms016 Exit_status=0 cput=00:03:53 mem=315968kb vmem=523488kb walltime=00:05:06
02/01/2008 03:58:07 cms016 Exit_status=0 cput=00:03:53 mem=321628kb vmem=532572kb walltime=00:05:56
02/01/2008 03:58:24 cms016 Exit_status=0 cput=00:04:18 mem=314292kb vmem=521984kb walltime=00:05:54
02/01/2008 03:59:08 cms016 Exit_status=0 cput=00:04:03 mem=343784kb vmem=553460kb walltime=00:07:05
02/01/2008 03:59:22 cms016 Exit_status=0 cput=00:04:12 mem=312004kb vmem=523540kb walltime=00:05:39
02/01/2008 03:59:22 cms016 Exit_status=0 cput=00:04:12 mem=311280kb vmem=513444kb walltime=00:05:19
02/01/2008 03:59:22 cms016 Exit_status=0 cput=00:04:18 mem=345664kb vmem=559620kb walltime=00:06:30
02/01/2008 04:00:52 cms016 Exit_status=0 cput=00:04:00 mem=359700kb vmem=596940kb walltime=00:08:07
02/01/2008 04:02:02 cms016 Exit_status=0 cput=00:03:55 mem=344100kb vmem=553676kb walltime=00:05:26
02/01/2008 04:02:27 cms016 Exit_status=0 cput=00:03:59 mem=332452kb vmem=523512kb walltime=00:06:29
02/01/2008 04:03:04 cms016 Exit_status=0 cput=00:04:11 mem=367872kb vmem=544052kb walltime=00:07:05
02/01/2008 04:03:19 cms016 Exit_status=0 cput=00:03:50 mem=348900kb vmem=534304kb walltime=00:06:10
02/01/2008 04:03:21 cms016 Exit_status=0 cput=00:04:17 mem=318676kb vmem=540216kb walltime=00:05:58
02/01/2008 04:03:34 cms016 Exit_status=0 cput=00:03:50 mem=330320kb vmem=549212kb walltime=00:07:29
02/01/2008 04:03:57 cms016 Exit_status=0 cput=00:03:52 mem=336112kb vmem=558368kb walltime=00:07:03
02/01/2008 04:04:45 cms016 Exit_status=0 cput=00:03:50 mem=332084kb vmem=549712kb walltime=00:06:41

02/01/2008 05:28:45 cms016 Exit_status=0 cput=00:03:48 mem=341348kb vmem=541540kb walltime=00:05:16

These are the jobs that Stefano Belforte ran between 2h and 5h (102 jobs). The walltimes are very short. Since we sampled the ganglia CE job information in 5 min intervals, most of these jobs were missed by our CE monitoring. The movers and I/O is measured at 1 minute intervals and therefore is more accurate.

02/01/2008 02:32:22 cms026 Exit_status=0 cput=00:01:07 mem=141136kb vmem=299408kb walltime=00:02:19
02/01/2008 02:32:50 cms026 Exit_status=0 cput=00:01:07 mem=139016kb vmem=293536kb walltime=00:01:47
02/01/2008 02:34:17 cms026 Exit_status=0 cput=00:01:06 mem=143448kb vmem=294276kb walltime=00:01:53
02/01/2008 02:34:36 cms026 Exit_status=0 cput=00:01:05 mem=129568kb vmem=280576kb walltime=00:01:43
02/01/2008 02:36:03 cms026 Exit_status=0 cput=00:01:06 mem=132140kb vmem=295188kb walltime=00:01:44
02/01/2008 02:36:19 cms026 Exit_status=0 cput=00:01:05 mem=117828kb vmem=262988kb walltime=00:01:41
02/01/2008 02:37:09 cms026 Exit_status=0 cput=00:01:06 mem=201516kb vmem=380664kb walltime=00:01:47
02/01/2008 02:37:48 cms026 Exit_status=0 cput=00:01:06 mem=123068kb vmem=272548kb walltime=00:01:43
02/01/2008 02:38:14 cms026 Exit_status=0 cput=00:01:06 mem=140612kb vmem=297896kb walltime=00:01:53
02/01/2008 02:38:20 cms026 Exit_status=0 cput=00:01:07 mem=138040kb vmem=273888kb walltime=00:01:57
02/01/2008 02:38:58 cms026 Exit_status=0 cput=00:01:06 mem=137188kb vmem=276152kb walltime=00:01:46
02/01/2008 02:39:45 cms026 Exit_status=0 cput=00:01:05 mem=135852kb vmem=291232kb walltime=00:01:56
02/01/2008 02:40:00 cms026 Exit_status=0 cput=00:01:07 mem=256220kb vmem=471876kb walltime=00:01:53
02/01/2008 02:40:01 cms026 Exit_status=0 cput=00:01:07 mem=122452kb vmem=273404kb walltime=00:01:44
02/01/2008 02:40:05 cms026 Exit_status=0 cput=00:01:07 mem=202388kb vmem=382480kb walltime=00:01:43
02/01/2008 02:40:39 cms026 Exit_status=0 cput=00:01:07 mem=130228kb vmem=288548kb walltime=00:01:52
02/01/2008 02:40:47 cms026 Exit_status=0 cput=00:01:05 mem=122472kb vmem=257516kb walltime=00:01:47
02/01/2008 02:40:51 cms026 Exit_status=0 cput=00:01:07 mem=201260kb vmem=398396kb walltime=00:01:49
02/01/2008 02:41:29 cms026 Exit_status=0 cput=00:01:06 mem=124236kb vmem=281116kb walltime=00:01:42
02/01/2008 02:41:37 cms026 Exit_status=0 cput=00:01:07 mem=119204kb vmem=254124kb walltime=00:01:45
02/01/2008 02:41:49 cms026 Exit_status=0 cput=00:01:06 mem=142028kb vmem=283768kb walltime=00:01:47
02/01/2008 02:41:49 cms026 Exit_status=0 cput=00:01:05 mem=142552kb vmem=293384kb walltime=00:01:47
02/01/2008 02:41:53 cms026 Exit_status=0 cput=00:01:06 mem=139184kb vmem=276460kb walltime=00:01:46
02/01/2008 02:42:24 cms026 Exit_status=0 cput=00:01:06 mem=200900kb vmem=392968kb walltime=00:01:43
02/01/2008 02:42:32 cms026 Exit_status=0 cput=00:01:07 mem=141716kb vmem=285288kb walltime=00:01:43
02/01/2008 02:42:36 cms026 Exit_status=0 cput=00:01:06 mem=140840kb vmem=300176kb walltime=00:01:44
02/01/2008 02:43:15 cms026 Exit_status=0 cput=00:01:06 mem=202824kb vmem=389060kb walltime=00:01:43
02/01/2008 02:43:33 cms026 Exit_status=0 cput=00:01:06 mem=138384kb vmem=281856kb walltime=00:01:54
02/01/2008 02:43:38 cms026 Exit_status=0 cput=00:01:05 mem=129552kb vmem=284744kb walltime=00:01:48
02/01/2008 02:43:38 cms026 Exit_status=0 cput=00:01:08 mem=129320kb vmem=267556kb walltime=00:01:48
02/01/2008 02:43:39 cms026 Exit_status=0 cput=00:01:08 mem=126504kb vmem=265172kb walltime=00:01:43
02/01/2008 02:44:07 cms026 Exit_status=0 cput=00:01:05 mem=138996kb vmem=290552kb walltime=00:01:42
02/01/2008 02:44:16 cms026 Exit_status=0 cput=00:01:05 mem=130880kb vmem=270148kb walltime=00:01:42
02/01/2008 02:44:20 cms026 Exit_status=0 cput=00:01:06 mem=130620kb vmem=283360kb walltime=00:01:42
02/01/2008 02:45:02 cms026 Exit_status=0 cput=00:01:06 mem=132800kb vmem=292896kb walltime=00:01:45
02/01/2008 02:45:19 cms026 Exit_status=0 cput=00:01:07 mem=123072kb vmem=261116kb walltime=00:01:45
02/01/2008 02:45:24 cms026 Exit_status=0 cput=00:01:07 mem=210964kb vmem=391088kb walltime=00:01:43
02/01/2008 02:45:25 cms026 Exit_status=0 cput=00:01:07 mem=200000kb vmem=377268kb walltime=00:01:45
02/01/2008 02:45:25 cms026 Exit_status=0 cput=00:01:07 mem=201508kb vmem=391476kb walltime=00:01:45
02/01/2008 02:45:53 cms026 Exit_status=0 cput=00:01:07 mem=130036kb vmem=281424kb walltime=00:01:43
02/01/2008 02:46:01 cms026 Exit_status=0 cput=00:01:07 mem=122336kb vmem=266100kb walltime=00:01:43
02/01/2008 02:46:10 cms026 Exit_status=0 cput=00:01:05 mem=202488kb vmem=399976kb walltime=00:01:48
02/01/2008 02:46:29 cms026 Exit_status=0 cput=00:01:07 mem=134652kb vmem=275156kb walltime=00:01:43
02/01/2008 02:46:46 cms026 Exit_status=0 cput=00:01:06 mem=123780kb vmem=276792kb walltime=00:01:42
02/01/2008 02:47:01 cms026 Exit_status=0 cput=00:01:06 mem=142708kb vmem=283144kb walltime=00:01:40
02/01/2008 02:47:16 cms026 Exit_status=0 cput=00:01:06 mem=137188kb vmem=295708kb walltime=00:01:49
02/01/2008 02:47:16 cms026 Exit_status=0 cput=00:01:07 mem=137264kb vmem=277232kb walltime=00:01:50
02/01/2008 02:47:39 cms026 Exit_status=0 cput=00:01:07 mem=201220kb vmem=390192kb walltime=00:01:44
02/01/2008 02:47:41 cms026 Exit_status=0 cput=00:01:07 mem=143292kb vmem=284204kb walltime=00:01:46
02/01/2008 02:47:41 cms026 Exit_status=0 cput=00:01:08 mem=137236kb vmem=275612kb walltime=00:02:16
02/01/2008 02:47:46 cms026 Exit_status=0 cput=00:01:06 mem=142272kb vmem=280476kb walltime=00:01:43

02/01/2008 04:00:03 cms026 Exit_status=0 cput=00:01:07 mem=132032kb vmem=274964kb walltime=00:01:57
02/01/2008 04:00:04 cms026 Exit_status=0 cput=00:01:06 mem=131052kb vmem=287828kb walltime=00:01:55
02/01/2008 04:00:14 cms026 Exit_status=0 cput=00:01:07 mem=123224kb vmem=262040kb walltime=00:01:48
02/01/2008 04:00:56 cms026 Exit_status=0 cput=00:01:07 mem=131304kb vmem=273296kb walltime=00:01:46
02/01/2008 04:00:56 cms026 Exit_status=0 cput=00:01:08 mem=124056kb vmem=261740kb walltime=00:01:45
02/01/2008 04:01:11 cms026 Exit_status=0 cput=00:01:07 mem=158392kb vmem=311712kb walltime=00:01:47
02/01/2008 04:01:15 cms026 Exit_status=0 cput=00:01:07 mem=143392kb vmem=285888kb walltime=00:01:51
02/01/2008 04:01:15 cms026 Exit_status=0 cput=00:01:08 mem=202524kb vmem=392832kb walltime=00:01:51
02/01/2008 04:01:47 cms026 Exit_status=0 cput=00:01:07 mem=201984kb vmem=381128kb walltime=00:01:43
02/01/2008 04:01:50 cms026 Exit_status=0 cput=00:01:05 mem=122436kb vmem=273036kb walltime=00:01:45
02/01/2008 04:01:54 cms026 Exit_status=0 cput=00:01:08 mem=201508kb vmem=378700kb walltime=00:01:46
02/01/2008 04:02:06 cms026 Exit_status=0 cput=00:01:05 mem=143052kb vmem=282720kb walltime=00:01:50
02/01/2008 04:02:45 cms026 Exit_status=0 cput=00:01:05 mem=140684kb vmem=277104kb walltime=00:01:48
02/01/2008 04:03:02 cms026 Exit_status=0 cput=00:01:06 mem=126236kb vmem=271332kb walltime=00:01:43
02/01/2008 04:03:14 cms026 Exit_status=0 cput=00:01:08 mem=138032kb vmem=290212kb walltime=00:02:20
02/01/2008 04:03:18 cms026 Exit_status=0 cput=00:01:07 mem=139396kb vmem=274660kb walltime=00:02:21
02/01/2008 04:03:36 cms026 Exit_status=0 cput=00:01:07 mem=125116kb vmem=266692kb walltime=00:02:22
02/01/2008 04:03:45 cms026 Exit_status=0 cput=00:01:08 mem=125624kb vmem=262068kb walltime=00:02:26
02/01/2008 04:03:49 cms026 Exit_status=0 cput=00:01:06 mem=139172kb vmem=295524kb walltime=00:01:57
02/01/2008 04:03:55 cms026 Exit_status=0 cput=00:01:07 mem=202500kb vmem=383580kb walltime=00:02:00
02/01/2008 04:03:59 cms026 Exit_status=0 cput=00:01:06 mem=137520kb vmem=280520kb walltime=00:01:54
02/01/2008 04:04:02 cms026 Exit_status=0 cput=00:01:07 mem=123196kb vmem=264920kb walltime=00:01:53
02/01/2008 04:04:03 cms026 Exit_status=0 cput=00:01:07 mem=135928kb vmem=278664kb walltime=00:02:15
02/01/2008 04:04:15 cms026 Exit_status=0 cput=00:01:07 mem=139920kb vmem=300028kb walltime=00:01:46
02/01/2008 04:04:16 cms026 Exit_status=0 cput=00:01:23 mem=143716kb vmem=297480kb walltime=00:02:14
02/01/2008 04:04:35 cms026 Exit_status=0 cput=00:01:06 mem=130212kb vmem=266488kb walltime=00:01:48
02/01/2008 04:04:58 cms026 Exit_status=0 cput=00:01:07 mem=200904kb vmem=380920kb walltime=00:01:51
02/01/2008 04:04:58 cms026 Exit_status=0 cput=00:01:07 mem=178580kb vmem=343696kb walltime=00:01:54
02/01/2008 04:05:06 cms026 Exit_status=0 cput=00:01:07 mem=138700kb vmem=295328kb walltime=00:01:49
02/01/2008 04:05:19 cms026 Exit_status=0 cput=00:01:07 mem=131196kb vmem=268384kb walltime=00:01:59
02/01/2008 04:05:21 cms026 Exit_status=0 cput=00:01:08 mem=129320kb vmem=262488kb walltime=00:01:59
02/01/2008 04:05:24 cms026 Exit_status=0 cput=00:01:07 mem=126640kb vmem=281112kb walltime=00:02:01
02/01/2008 04:05:36 cms026 Exit_status=0 cput=00:01:07 mem=201644kb vmem=395020kb walltime=00:02:00
02/01/2008 04:05:37 cms026 Exit_status=0 cput=00:01:06 mem=122004kb vmem=265760kb walltime=00:01:58
02/01/2008 04:05:39 cms026 Exit_status=0 cput=00:01:07 mem=200712kb vmem=398752kb walltime=00:02:01
02/01/2008 04:05:42 cms026 Exit_status=0 cput=00:01:06 mem=202460kb vmem=394348kb walltime=00:01:52
02/01/2008 04:05:42 cms026 Exit_status=0 cput=00:01:05 mem=144344kb vmem=284460kb walltime=00:01:55
02/01/2008 04:05:46 cms026 Exit_status=0 cput=00:01:06 mem=202004kb vmem=381016kb walltime=00:01:50
02/01/2008 04:05:48 cms026 Exit_status=0 cput=00:01:07 mem=233600kb vmem=385044kb walltime=00:01:46
02/01/2008 04:05:48 cms026 Exit_status=0 cput=00:01:05 mem=140708kb vmem=273252kb walltime=00:01:44
02/01/2008 04:05:53 cms026 Exit_status=0 cput=00:01:07 mem=138324kb vmem=284432kb walltime=00:01:47
02/01/2008 04:06:03 cms026 Exit_status=0 cput=00:01:06 mem=130956kb vmem=280700kb walltime=00:01:45
02/01/2008 04:06:16 cms026 Exit_status=0 cput=00:01:07 mem=140608kb vmem=298832kb walltime=00:02:16
02/01/2008 04:06:26 cms026 Exit_status=0 cput=00:01:06 mem=202432kb vmem=380592kb walltime=00:01:49
02/01/2008 04:06:30 cms026 Exit_status=0 cput=00:01:24 mem=126388kb vmem=284900kb walltime=00:02:11
02/01/2008 04:06:34 cms026 Exit_status=0 cput=00:01:08 mem=138852kb vmem=293412kb walltime=00:01:47
02/01/2008 04:06:44 cms026 Exit_status=0 cput=00:01:08 mem=131404kb vmem=281508kb walltime=00:01:44
02/01/2008 04:06:49 cms026 Exit_status=0 cput=00:01:08 mem=132808kb vmem=277248kb walltime=00:01:50
02/01/2008 04:06:50 cms026 Exit_status=0 cput=00:01:09 mem=129736kb vmem=287832kb walltime=00:01:41
02/01/2008 04:07:04 cms026 Exit_status=0 cput=00:01:07 mem=202492kb vmem=380740kb walltime=00:01:41
02/01/2008 04:07:10 cms026 Exit_status=0 cput=00:01:06 mem=202096kb vmem=375400kb walltime=00:01:46

02/01/2008 05:51:03 cms026 Exit_status=0 cput=00:01:07 mem=130608kb vmem=288768kb walltime=00:01:45


Job numbers for V. Miccio and S. Belforte are 150 and 204 as reported by the dashboard, and 50 and 102 reported from the gatekeeper logs. It is curious that the numbers are almost exactly trebled/doubled. Could there be some double counting?

Julia Andreeva from ARDA/CMS explained, that this query (with the checkbox "all jobs regardless" enabled) actually does not take the second entered time value into account at all. It just counts the jobs from the first time value to the present. This should be explained better on the dashboard Web-UI.

Arrow left Go to previous page / next page of CMS site log MOVED TO...

-- DerekFeichtinger - 01 Feb 2008

Topic attachments
I Attachment History Action Size Date Who Comment
JPEGjpg CSCS_CMS_IO-20080203.jpg r1 manage 42.3 K 2008-02-01 - 09:08 DerekFeichtinger  
JPEGjpg CSCS_CMSmovers-20080202.jpg r1 manage 16.3 K 2008-02-01 - 09:07 DerekFeichtinger  
JPEGjpg CSCS_jobrobot-20080201.jpg r1 manage 48.2 K 2008-02-01 - 09:07 DerekFeichtinger  
JPEGjpg CSCS_jobrobotB-20080201.jpg r1 manage 52.6 K 2008-02-01 - 10:13 DerekFeichtinger  
JPEGjpg CSCS_jobs-20080204.jpg r1 manage 32.7 K 2008-02-01 - 09:07 DerekFeichtinger  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2008-03-28 - DerekFeichtinger
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback