MeetingSwissGridOperations20160707 < LCGTier2

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.EgiGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.ChippComputingBoardGroup
-->

---+ Swiss Grid Operations Meeting on 2016-07-07 at 14:00
   * *Place*: Vidyo (room: Swiss_Grid_Operations_Meeting, extension: 10537598)
   * *External link*: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=FAEn4zjAba7BqoQ11TGZu66VSDE
   * *Phone gate*: From Switzerland: 0227671400 (portal) + 10537598 (extension) + # (pound sign)
   * *IRC chat*: irc:gridchat.cscs.ch:994#lcg (ask pw via email)
   * *Switch Vidyo SIP IP*: 137.138.248.204
%TOC%

---++ Site status
---+++ CSCS
   * Some accounting numbers:<br /> <table cellspacing="0" border="0"> <colgroup width="70"></colgroup> <colgroup width="112" span="2"></colgroup> <colgroup width="58"></colgroup> <colgroup width="84"></colgroup> <colgroup width="185"></colgroup> <colgroup width="130"></colgroup> <colgroup width="85" span="2"></colgroup> <tbody> <tr> <td align="left" height="17">account</td> <td align="left">% num jobs</td> <td align="left">% of wall</td> <td align="left">count(*)</td> <td align="left">walltime sec</td> <td align="left">sum(round(max_vsize/1024))</td> <td align="left">sum_tres_req_mem</td> <td align="left">mem_diff</td> <td align="left">%</td> </tr> <tr> <td align="left" height="17">total:</td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> <tr> <td align="left" height="17">atlas</td> <td align="right">100.00%</td> <td align="right">100.00%</td> <td align="right">288913</td> <td align="right">2694614271</td> <td align="right">1,617,843,841</td> <td align="right">1,389,126,500</td> <td align="right">228,717,341</td> <td align="right">116.46%</td> </tr> <tr> <td align="left" height="17">cms</td> <td align="right">100.00%</td> <td align="right">100.00%</td> <td align="right">50840</td> <td align="right">1535630187</td> <td align="right">230,934,497</td> <td align="right">356,035,968</td> <td align="right">-125,101,471</td> <td align="right">64.86%</td> </tr> <tr> <td align="left" height="17">lhcb</td> <td align="right">100.00%</td> <td align="right">100.00%</td> <td align="right">57574</td> <td align="right">3211019505</td> <td align="right">255,594,384</td> <td align="right">115,148,000</td> <td align="right">140,446,384</td> <td align="right">221.97%</td> </tr> <tr> <td align="left" height="17"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> <tr> <td align="left" height="17"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> <tr> <td align="left" height="17">req&lt;=2000:</td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> <tr> <td align="left" height="17">atlas</td> <td align="right">68.50%</td> <td align="right">43.09%</td> <td align="right">197903</td> <td align="right">1160991397</td> <td align="right">547,848,230</td> <td align="right">386,762,000</td> <td align="right">161,086,230</td> <td align="right">141.65%</td> </tr> <tr> <td align="left" height="17">cms</td> <td align="right">74.38%</td> <td align="right">0.28%</td> <td align="right">37816</td> <td align="right">4244836</td> <td align="right">30,376,806</td> <td align="right">75,632,000</td> <td align="right">-45,255,194</td> <td align="right">40.16%</td> </tr> <tr> <td align="left" height="17">lhcb</td> <td align="right">100.00%</td> <td align="right">100.00%</td> <td align="right">57572</td> <td align="right">3210873808</td> <td align="right">255,585,171</td> <td align="right">115,144,000</td> <td align="right">140,441,171</td> <td align="right">221.97%</td> </tr> <tr> <td align="left" height="17"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> <tr> <td align="left" height="17"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> <tr> <td align="left" height="17">req&gt;2000:</td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> <tr> <td align="left" height="17">atlas</td> <td align="right">31.50%</td> <td align="right">56.91%</td> <td align="right">91007</td> <td align="right">1533609961</td> <td align="right">1,069,984,255</td> <td align="right">1,002,358,500</td> <td align="right">67,625,755</td> <td align="right">106.75%</td> </tr> <tr> <td align="left" height="17">cms</td> <td align="right">25.62%</td> <td align="right">99.72%</td> <td align="right">13024</td> <td align="right">1531385351</td> <td align="right">200,557,691</td> <td align="right">280,403,968</td> <td align="right">-79,846,277</td> <td align="right">71.52%</td> </tr> <tr> <td align="left" height="17"><br /></td> <td align="right">0.00%</td> <td align="right">0.00%</td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> <td align="left"><br /></td> </tr> </tbody> </table> <br /><br />
   * Accounting numbers (from scheduler) from last month

---+++ PSI
   * Upgraded my 2 HP CentOS7 NFSv4 NAS to [[http://zfsonlinux.org/][ZoL v0.6.5.7]] 
      * 1st NAS is the primary featuring 24 SAS disks 15k 600GB
      * 2ns NAS is the secondary featuring 12 SATA disks 7.2k 3000GB ( cold backup )
      * both feature a dual 10Gb/s card put in LACP bonding mode
* dCache on ZoL <ins> * on the secondary NAS I'm going to make a ZFS fs for dCache and provide ~5TB to the PSI T3 ; it's a shame to use this HW only for backups ( 5y warranty ) </ins> * again on the secondary NAS I made ZFS fs for dCache : <ins> * [[http://t3mon.psi.ch/ganglia/PSIT3-custom/accounting.txt][Accounting numbers (from scheduler) from last month]] </ins> * <pre>[root@t3nfs02 ~]# zfs list -d1
NAME                    USED  AVAIL  REFER  MOUNTPOINT
<ins>---+++ UNIBE-LHEP
</ins>data01                 1.33T  9.15T  32.0K  /zfs/data01
data01/dcache           100G  9.15T  32.0K  %BLUE%/zfs/data01/dcache%ENDCOLOR%
<ins>*Operations*
</ins>data01/t3nfs01_data01  1.23T  9.15T  32.0K  /zfs/data01/t3nfs01_data01
<ins>   * tough month: several issues with full root partitions on wn's and one lustre oss not working well. Also the cloud cluster didn't perform too well (didn't follow-up with SWITCH yet)
</ins>data02                 4.33T  6.15T  32.0K  /zfs/data02
<ins>*ATLAS specific operations*
</ins>data02/dcache           100G  6.15T  32.0K  %BLUE%/zfs/data02/dcache%ENDCOLOR%
<ins>   * 
</ins>data02/t3nfs01_data01  4.23T  6.15T  32.0K  /zfs/data02/t3nfs01_data01
<ins>      * ICHEP conference in August =&gt; steep rise in analysis jobs (lustre suffers)
</ins></pre> <ins> * One user jobs very instrumental in killing the shared file system. Could not discover exactly what was wrong with these and had not the time to follow up, so ended up bannign analysis temporarily </ins> * dCache tuning <ins> * Also plenty of data intensive prod workloads (mainly derivations) runnign concurrently (lustre suffers more) </ins> * <pre>[root@t3se01 layouts]# grep max /etc/dcache/layouts/t3se01.conf 
<ins>      * Issue with some event generation workloads (madgraph) writing large files in /tmp. Root are too small On SunBlade nodes to absorbe that, even with a very aggressive cleanup cron job. Ended up having to ban evegen+simulation from the site as a temporary measure!
</ins>srm.request.max-requests=400
<ins>*HammerCloud report [1]*
</ins>srm.request.put.max-requests=100
<ins>   * UNIBE-LHEP online &gt;79% (last month). Reflects the instabilities mentioned above
</ins>srm.request.get.max-inprogress=100
<ins>   * UNIBE-ID 99%
</ins>srm.request.copy.max-inprogress=100
<ins>   * UNIBE-LHEP_CLOUD* 71% 
</ins>srm.request.max-transfers=100
<ins>[1] http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteviewhistorywithstatistics?columnid=562&view=Shifter%20view#time=720&start_date=&end_date=&use_downtimes=false&merge_colors=false&sites=multiple&clouds=ND&site=UNIBE-LHEP,UNIBE-LHEP-UBELIX,UNIBE-LHEP-UBELIX_MCORE,UNIBE-LHEP_CLOUD,UNIBE-LHEP_CLOUD_MCORE,UNIBE-LHEP_MCORE
</ins></pre> * [[http://t3mon.psi.ch/ganglia/PSIT3-custom/accounting.txt][Accounting numbers (from scheduler) from last month]] <ins>*ATLAS resource delivery [2]* </ins> ---+++ UNIBE-LHEP <ins> * All jobs: 56% of ATLAS/CH (WallTime), 77% of ATLAS/CH (CPUtime) </ins> * Xxx <ins> * Good jobs: 69% of ATLAS CH (WallTime), 79% of ATLAS/CH (CPUtime) </ins> * Accounting numbers (from scheduler) from last month <ins>[2] http://dashb-atlas-job-prototype.cern.ch/dashboard/request.py/dailysummary#button=cpuconsumption&sites%5B%5D=CSCS-LCG2&sites%5B%5D=UNIBE-LHEP&sitesCat%5B%5D=All+Countries&resourcetype=All&sitesSort=2&sitesCatSort=0&start=2016-06-01&end=2016-06-30&timerange=daily&granularity=Monthly&generic=0&sortby=0&series=All </ins> ---+++ UNIBE-ID <ins> * *Accounting numbers (from scheduler) from last month (Jun 2016)* ( includes ce03/CLOUD ) </ins> * Mostly smooth operation <ins> * WC h: 960084 (ATLAS) - 1172 (t2k.org) - 1104 (uboone) - 16 (ops) </ins> * Procurement: <ins> * *Accounting numbers (from ATLAS dashboard) from last month* (Jun 2016) </ins> * 80 new server (76*20 + 4*16 =&gt; 1584 new cores; disontinued 144 cores (oldest nodes) <ins> * CPU h: 858693 (May value: 1194137) </ins> * installed and provisioned <ins> * WC h: 1057196 (May value: 1358408) </ins> * Migration from OGSGE =&gt; Slurm planned for Q4 * Probs with NAMD jobs (using ibverbs directly) =&gt; low level IB errors from mlx4 regarding qp <ins>---+++ UNIBE-ID </ins> * no errors with MPI jobs using ompi or the like <ins> * Xxx </ins> * no errors with storage (GPFS over RDMA)
   * ATLAS specific: large number of random a-rex crashes within the last 2 weeks 
      * reason unknown, happened 24x between 2016-06-15 and last monday; no crash since 3 days

---+++ UNIGE

* Operations <ins> * Xxx </ins> * 10 machines added into the batch system (80 cores) + 3 machines replaced: <ins> * Accounting numbers (from scheduler) from last month </ins> * DELL - Intel Xeon @ 2.4 GHz - with 8 cores and 48 GB of memory
   * 
      * RAID controller: Common problem for our DPM and NFS File servers (It happened like 3/4 times during last months)
      * Increased activity from DPNC users to run in the batch system (other groups, in addition to ATLAS)
      * Still not in ATLAS production, problems related with memory (hints provided by Gianfranco)
   * Data Management: 
      * User datasets from UniGe for ATLASLOCALGROUPDISK at CSCS deleted (space can be moved to ATLASSCRATCHDISK)
      * Some problems for central deletion (fixed) - permissions related: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122024
   * [[%ATTACHURL%/g07.2016.06.log][Accounting numbers (from scheduler) from last month]]

---+++ NGI_CH
   * Xxx
   * NGI-CH Open Tickets review

---++ Other topics
   * Topic1
   * Topic2
Next meeting date:

---++ A.O.B.

---++ Attendants
   * CSCS:
   * CMS:
* ATLAS: Michael Rolli (UNIBE-ID) =&gt; absent being ill, nevertheless some text above <ins> * ATLAS: </ins> * LHCb:
   * EGI:

---++ Action items
   * Item1
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	Screen_Shot_2016-07-07_at_14.04.57.png	r1	manage	76.4 K	2016-07-07 - 12:08	GianfrancoSciacca	Memory request summary UNIBE-LHEP
log	g07.2016.06.log	r1	manage	1.1 K	2016-07-07 - 11:05	LuisMarch	Accounting UniGe June 2016
This topic: LCGTier2 > WebHome > MeetingsBoard > MeetingSwissGridOperations20160707
Topic revision: r10 - 2016-07-07 - DinoConciatore