MeetingSwissGridOperations20160512 < LCGTier2

Tags: view all tags
<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.EgiGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.ChippComputingBoardGroup
-->

---+ Swiss Grid Operations Meeting on 2016-05-12 at 14:00
   * *Place*: Vidyo (room: Swiss_Grid_Operations_Meeting, extension: 109305236)
   * *External link*: http://vidyoportal.cern.ch/flex.html?roomdirect.html&key=gDf6l4RlIAGN
   * *Phone gate*: From Switzerland: 0227671400 (portal) + 109305236 (extension) + # (pound sign)
   * *IRC chat*: irc:gridchat.cscs.ch:994#lcg (ask pw via email)
   * *Switch Vidyo SIP IP*: 137.138.248.204
%TOC%

---++ Site status
---+++ CSCS

<strong><span style="background-color: transparent;">SYSTEM</span><br /></strong>
   * <span style="background-color: transparent;">IB eth bridges replaced</span>
*CONFLICT* original 8:

<span style="background-color: transparent;"> *SYSTEM* </span>

*CONFLICT* version new: <span style="background-color: transparent;"> *SYSTEM* </span> <ins> * Accounting numbers (from scheduler) from last month </ins> * IB eth bridges replaced

*CONFLICT* end
   * A few IB QDR switches replaced with FDR switches
   * Compute nodes re-installed 
      * CSCS Central puppet
      * CSCS LDAP for users
      * CSCS NFS for home
   * ARC CE fresh installed with the following queues: 
      * arc01 64nodes (sandy bridge nodes, 64GB ram, 32 cores)
      * arc02 48 nodes (ivy bridge nodes, 128GB ram, 40 cores)
      * arc03 40 nodes (haswell nodes, 128GB ram, 48 cores) soon updated to v4
   * CREAM01/02/03: reviewing accounting before final shutdown
   * All Virtual machines are running on CSCS central VMware
   * CMS re-installation to be planned with Puppet base installation (Firewall, Users, Grid Certificates, ..)
   * Current allocation over 90% 
      * Allocation problems on the old 64GB RAM nodes (arc01 queue)
*STORAGE*
<div id="_mcePaste">
   * <span style="background-color: transparent;"> __GPFS__ </span> 
      * <span style="background-color: transparent;">Few weeks ago we reached 310M of used inodes on the scratch fileset</span>
      * <span style="background-color: transparent;">servers high load -&gt; slow cleaning policy -&gt; job problems</span>
      * <span style="background-color: transparent;">filesystem stayed online</span>
</div> <div style="padding-left: 30px;" id="_mcePaste">Consequence</div> <div id="_mcePaste">
   * 
      * <span style="background-color: transparent;">per user inode quota (50M)</span>
      * <span style="background-color: transparent;">inode usage alerts<br /><br /></span>
   * __dCache__ 
      * <span style="background-color: transparent;">9K+ active connections (record?)</span>
      * <span style="background-color: transparent;">12+GBit internet bandwith measured on the network (2x 6+Gbit)</span>
      * <span style="background-color: transparent;">Real limit was about 2x8Gbit</span>
      * <span style="background-color: transparent;">New limit is about 80Gbit with the new gateways<br /></span>
<img width="1249" alt="dcache connections" src="https://wiki.chipp.ch/twiki/pub/LCGTier2/MeetingSwissGridOperations20160512/poolqueueplots2.png" title="dcache connections" height="873" />

</div> <div style="padding-left: 30px;" id="_mcePaste"><span style="background-color: transparent;">Some technical details and numbers </span>of the new storage that will be available in the next days (1PB)</div>

   1 <span style="background-color: transparent;"><strong>NETAPP (0.5PB)</strong><br /><span style="font-weight: normal;">2xController / 4xFC16 Links / 10xLUNs / 12x6TB drives per LUN (RAID6)</span><span style="white-space: pre; background-color: transparent;"> </span><br /><em><br />Architecture<br /><img width="1171" alt="cscs san" src="https://wiki.chipp.ch/twiki/pub/LCGTier2/MeetingSwissGridOperations20160512/phoenix-cscs-san.png" title="cscs san" height="763" /><br /><br /><br />Performances (6x dd from 2 servers each on a different LUN, 3x controller) <br /></em><img width="582" alt="e5600" src="https://wiki.chipp.ch/twiki/pub/LCGTier2/MeetingSwissGridOperations20160512/e5600two.png" title="e5600" height="850" /><br /><br /></span>
   1 <strong style="background-color: transparent;">DDN SFA12K (0.5PB)<br /></strong>2x Controller / 4x Storage Processors / 4x FC16 Links (1xStorage Processor) / 24LUNs / 10x3TB drives per LUN (RAID6)<span style="white-space: pre; background-color: transparent;"> </span><br /><br /><em>Architecture</em> -&gt; same CSCS integration<br /><em>Performances</em> will follow asap

---+++ PSI
   * *T3 upgraded to 10Gbs* 
      * The local Net team deployed a CISCO Extender, 32 ports 10Gbs CAT6, 8 uplinks 10Gbs in Fibre ; so far 4/8 uplinks cabled
      * The 32 ports can be organized as [[https://en.wikipedia.org/wiki/Link_aggregation#Link_Aggregation_Control_Protocol][LACP]] groups ; we made 5 groups with 2 ports inside ( Tot. 10 ports cabled ) ; both the dCache Storage servers and the ZFS/NFSv4 NAS are connected to these LACP groups
      * Other 9 ports are used for the latest 10Gbs WNs ( ~500 CPUs core )
      * Generally speaking, it was a nice 1Gbs to 10Gbs transition without unexpected troubles
   * *dCache upgraded from 2.13 to 2.15* 
      * Configuration files as in 2.13 worked also in [[https://www.dcache.org/manuals/Book-2.15/Book-fhs.shtml][2.15]], nice
      * But the Chimera Tables changed ; there is a new field [[https://raw.githubusercontent.com/dCache/dcache/552fc76f99148b6721bef55f0ffa28e6eb32a439/modules/chimera/src/main/resources/org/dcache/chimera/changelog/changeset-2.15.xml][inumber]] that's used as a key here and there in respect of =ipnfsid= age ( that's still in the tables though )
      * The Chimera Table change means that both [[http://www.desy.de/~leffhalm/chimera-dump.html][chimera-dump]] and my materialized view [[https://bitbucket.org/fabio79ch/v_pnfs/wiki/Home][v_pnfs]] don't work right now
      * I won't update [[http://www.desy.de/~leffhalm/chimera-dump.html][chimera-dump]] while I'm updating [[https://bitbucket.org/fabio79ch/v_pnfs/wiki/Home][v_pnfs]] to be 2.15 compatible ; CSCS will have my same issues on its own dCache roadmap
   * *Need to distinguish between NFSv4 interactive traffic ( users ) and batch system traffic ( jobs )* 
      * Because we've added ~500 CPUs core sometime the hundreds of jobs make our NFSv4 slow ( &gt;1s delay ) and the users working interactively on the UIs get affected ; we need to assign more 'NFSv4 priority' to these UIs in respect of the WNs, for instance by setting [[http://shorewall.net/simple_traffic_shaping.html][shorewall]] on the NFSv4 server ? if you have already solved similar issues kindly contact us
   * [[http://t3mon.psi.ch/ganglia/PSIT3-custom/accounting.txt][Accounting numbers (from scheduler) from last month]]

---+++ UNIBE-LHEP

*Operations*
   * <span style="background-color: transparent;">stable, no incidents to report</span>
*ATLAS specific operations*
   * HC online ~90% (last month). Still room for improvement, btu not too big impact since interruptions are not long enough to cause the site to drain. UNIBE-ID &gt;96%
<span style="font-family: arial, verdana, sans-serif; font-size: 14px; background-color: transparent;">http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteviewhistorywithstatistics?columnid=562&view=Shifter%20view#time=720&start_date=&end_date=&use_downtimes=false&merge_colors=false&sites=multiple&clouds=ND&site=UNIBE-LHEP,UNIBE-LHEP-UBELIX,UNIBE-LHEP-UBELIX_MCORE,UNIBE-LHEP_CLOUD,UNIBE-LHEP_CLOUD_MCORE,UNIBE-LHEP_MCORE</span>

<span style="font-family: arial, verdana, sans-serif; font-size: 14px; background-color: transparent;"><br /></span>
   * 52% of ATLAS/CH WT, 54% CPUtime in April
   * No progress on DPM head node migration to SLC6 and ATLAS storage dumps

   * <span style="background-color: transparent;"> *Accounting numbers (from scheduler) from last month (Apr 2016)* ( includes ce03/CLOUD )</span> 
      * <span style="background-color: transparent;">WC h: 1028684 (ATLAS) - 149450 (t2k.org) -10776 (uboone) - 11 (ops)</span>
   * *Accounting numbers (from ATLAS dashboard) from last month* (Apr 2016) 
      * CPU h: 967738
      * WC h: 1108219
---+++ UNIBE-ID
   * Smooth operation the last weeks, except 
      * a lot of CERN jobs get killed due to h_vmem limit violation 
         * known issue: gridengnine counting issues with shared libraries
         * no patch for (O)SGE available
         * was no problem in the past (xxxx - 2015)
         * no solution known (except moving to SLURM - not possible ATM); running 32-bit? only sgemaster? Experiences?
   * Procueremnt: 
      * 76 new compute nodes (E5v4-10C@2.2GHz) ordered and get devlivered on 9th+10th of June
      * doubling IB-Spine Switches =&gt; recabling of whole IB stuff

---+++ UNIGE
   * Operations: 
      * Running smoothly with higher user usage of the cluster for last months 
         * <span style="color: #0000ff; text-decoration: underline;">[[https://wiki.chipp.ch/twiki/pub/LCGTier2/MeetingSwissGridOperations20160512/UniGe_GRID_last_month.png][UniGe GRID last month ganglia]]</span>
         * <span style="color: #0000ff; text-decoration: underline;">[[https://wiki.chipp.ch/twiki/pub/LCGTier2/MeetingSwissGridOperations20160512/UniGe_Users_last_month.png][UniGe Users last month ganglia]]</span>
      * 2 NFS File Servers for DPM SE with RAID controller damage: Both changed and came back into production
      * ATLAS production jobs stopped since last Sunday: Contacted Gianfranco to ask about it 
         * Some problems with Data Management at some sites, but we were removed and not put it back until today
      * A long list of action items: 
         * Such as CentOS,
         * add WNs into the batch system,
         * add new NFS File Server for ATLAS DPM SE,
         * create a new pool inside DPM SE for another group: DAMPE
         * Network upgrade to 10 Gb/s
         * Move to SLURM for batch system and Puppet for DM SE
   * Accounting numbers (from scheduler) from last month 
      * [[%ATTACHURL%/g07.201603.log][g07.201603.log]]
      * [[%ATTACHURL%/g07.201604.log][g07.201604.log]]

---+++ NGI_CH

* Xxx <ins> * EGI mostly focussing on Fed Cloud operation consolidation </ins> * NGI-CH Open Tickets review <ins> * NGI-CH Open Tickets review: https://ggus.eu/index.php?mode=ticket_search&supportunit=NGI_CH&status=open&timeframe=any&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO </ins> * No ticket not touched for 1 week or more

*CONFLICT* original 8: <span style="background-color: transparent;"> *SYSTEM* </span> * Xxx <ins> * EGI mostly focussing on Fed Cloud operation consolidation </ins> * NGI-CH Open Tickets review <ins> * NGI-CH Open Tickets review: https://ggus.eu/index.php?mode=ticket_search&supportunit=NGI_CH&status=open&timeframe=any&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO </ins> * No ticket not touched for 1 week or more

*CONFLICT* version 9:

   * 
*CONFLICT* version new: <span style="background-color: transparent;"> *SYSTEM* </span> <ins> * Accounting numbers (from scheduler) from last month </ins> * IB eth bridges replaced
   * <ins>EGI mostly focussing on Fed Cloud operation consolidation</ins>
   * <ins>New MW products for CenOS 7: </ins>
   * ARC
   * Argus
   * dcache
*CONFLICT* end
   * fts3
   * site-bdii
   * top-bdii
   * <ins>NGI-CH Open Tickets review: https://ggus.eu/index.php?mode=ticket_search&supportunit=NGI_CH&status=open&timeframe=any&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO</ins> 
      * <span style="background-color: transparent;">No ticket not touched for 1 week or more</span>
      * <span style="background-color: transparent;">Changed 2 tickets from "assigned" state to "in progress"</span>
---++ Other topics
   * Topic1
   * Topic2
Next meeting date:

NOTE: Week of 30th May is the Nordugrid conference (GS not available)

---++ A.O.B.

---++ Attendants
   * CSCS:
   * CMS: Fabio Martinelli
* ATLAS: <ins> * ATLAS: Apologies (Gianfranco) </ins> * LHCb: * EGI: <ins> * EGI: Apologies (Gianfranco) </ins>
---++ Action items
   * Item1
%TWISTY%
   * dcache 9k: <br /> <img width="1249" alt="poolqueueplots2.png" src="%ATTACHURLPATH%/poolqueueplots2.png" height="873" />

   * e5600perf: <br /> <img width="582" alt="e5600two.png" src="%ATTACHURLPATH%/e5600two.png" height="850" />

   * sancscs: <br /> <img width="1171" alt="phoenix-cscs-san.png" src="%ATTACHURLPATH%/phoenix-cscs-san.png" height="763" />
%ENDTWISTY%
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	May12_CMS.pdf	r1	manage	1650.8 K	2016-05-12 - 12:00	JoosepPata	CMS report on computing resources
png	UniGe_GRID_last_month.png	r1	manage	41.6 K	2016-05-12 - 12:12	LuisMarch	Unige GRID last month
png	UniGe_Users_last_month.png	r1	manage	33.3 K	2016-05-12 - 12:13	LuisMarch	UniGe Users last month
png	e5600two.png	r1	manage	56.7 K	2016-05-12 - 10:08	DarioPetrusic	e5600perf
log	g07.201603.log	r1	manage	1.2 K	2016-05-12 - 12:09	LuisMarch	Unige accounting - March 2016
log	g07.201604.log	r1	manage	1.1 K	2016-05-12 - 12:10	LuisMarch	UniGe accounting - April 2016
png	phoenix-cscs-san.png	r1	manage	95.1 K	2016-05-12 - 10:08	DarioPetrusic	sancscs
png	poolqueueplots2.png	r1	manage	24.8 K	2016-05-12 - 10:04	DarioPetrusic	dcache 9k