MeetingSwissGridOperations20181206 < LCGTier2

Tags: view all tags
&lt;!-- keep this as a security measure:<br /> * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.EgiGroup<br /> * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup<br /> #uncomment this if you want the page only be viewable by the internal people<br /> #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.ChippComputingBoardGroup<br />--&gt;

---+ Swiss Grid Operations Meeting on 2018-12-06 at 14:00
   * *Place*: Vidyo (room: Swiss_Grid_Operations_Meeting, extension: 10537598)
   * *External link*: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=FAEn4zjAba7BqoQ11TGZu66VSDE
   * *Phone gate*: From Switzerland: 0227671400 (portal) + 10537598 (extension) + # (pound sign)
   * *IRC chat*: irc:gridchat.cscs.ch:994#lcg (ask pw via email)
   * *Switch Vidyo SIP IP*: 137.138.248.204
%TOC%

---++ Site status
---+++ CSCS

*System*
   * Started planning services transition to Piz Daint. Question about the new ARC CE version to Gianfranco
   * Issues with some nodes and slurm subnet, reconfiguration needed.
   * Grafana dashboard is ready to be exposed (data sync in progress)
*dCache*
   * Process to IPv6 / dual-stack completed. Required quite some effort
   * We need to add dual-stack to CMS02 too
   * Updated to 3.2.40
   * Planning next year data migration (due to a complete storage renewal)
   * Planning head nodes (re)installation

*Scratch*
   * Stable operations
   * Much improved IO between (Daint) DVS nodes and storage nodes since the last DVS upgrade
   * Outperforming SSD cache nominal performance (*see attachment "ssd-cache-perf.png*)"
   * We should have a further improvement with the new software

---+++ PSI
   * new server installation for !NetApp
   * re-installation of admin node
   * [[http://t3mon.psi.ch/PSIT3-custom/accounting.txt][Accounting numbers (from scheduler) from last month]]

---+++ UNIBE-LHEP
   * 
   * Stable operation, slightly lower delivery for LHEP (dying nodes). Pledged: 18k, delivered 20.6k
   * Ubelix back on 8th November
   * Running an average &gt;2100 slots (&lt;1900 last month, 2500 typical), Ubelix back to 23% (typical)
   * <table border="1" cellpadding="0" cellspacing="1"> <tbody> <tr><th> </th><th> </th><th> </th> <td> </td> <td> </td> </tr> <tr> <td> </td> <td> </td> <td> </td> <td> </td> <td> </td> </tr> <tr> <td> </td> <td> </td> <td> </td> <td> </td> <td> </td> </tr> <tr> <td> </td> <td> </td> <td> </td> <td> </td> <td> </td> </tr> <tr> <td> </td> <td> </td> <td> </td> <td> </td> <td> </td> </tr> </tbody> </table> <p> </p> <p> </p> <p> </p> *Accounting numbers (from scheduler) from last month (October), LHEP only* <p> </p> %EDITTABLE{}% <p> </p> <p> </p>
| *VO* | *Job Type* | *Produced WC core-hours* | | |
| ATLAS | Any | 1102994 (1157991 in Oct) | | |
| ops | Any | 46 (44 in Oct) | | |
| t2k.org | Any | 0 | | |
| uboone | Any | 0 | | |
 <img alt="" height="262" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/consumptions_individual?sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-11-01&end=2018-11-30&timeRange=daily&granularity=Monthly&generic=0&sortBy=16&series=All&type=ewa" width="350" /><br /><br />

 <b>6-month history Unibe (pledge: 18 kHS06)<br /><img alt="" height="256" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/resourceutilization_individual?sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-06-01&end=2018-11-30&timeRange=daily&granularity=Monthly&generic=0&sortBy=16&diag1=0&diag2=0&diag3=0&diag4=0&diag5=0&diag6=0&diag7=0&diag8=0&diagT=0&diag8pl=0&series=All&type=wchs" width="341" /><br /><br /></b>
   * <b>Swiss ATLAS statistics<br /><br /></b>
      * *HC availability*
      * could not retrieve data
      * *Runnins slots*
      * CSCS: 3000 (3300 October) ; UniBe: 2150 (1850 October)
      * *Accounting Numbers from ATLAs dashboard (November) CSCS+UniBe* <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p>

 <br />%EDITTABLE{}%

| *Cluster* | *Job Type* | *Produced WC core-hours* | *Good vs Bad WC %* | *CPU eff good jobs %* |
| CSCS | Any | 2397014; 63% (Oct: 2901550 69%) | 0.79 (Oct: 0.71) | 0.85 (Oct: 0.89) |
| UniBe | Any | 1411100; 37% (Oct: 1266896 31%) | 0.84 (Oct: 0.85) | 0.83 (Oct: 0.85) |
 <img alt="" height="243" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/resourceutilization_individual?sites=CSCS-LCG2&sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-11-01&end=2018-11-30&timeRange=daily&granularity=8 Hours&generic=0&sortBy=0&diag1=0&diag2=0&diag3=0&diag4=0&diag5=0&diag6=0&diag7=0&diag8=0&diagT=0&diag8pl=0&series=All&type=wchs" width="324" /><img alt="" height="221" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/resourceutilization_individual?sites=CSCS-LCG2&sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-11-01&end=2018-11-30&timeRange=daily&granularity=Hourly&generic=0&sortBy=0&diag1=0&diag2=0&diag3=0&diag4=0&diag5=0&diag6=0&diag7=0&diag8=0&diagT=0&diag8pl=0&series=All&type=a" width="293" /><br /><img alt="" height="213" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/consumptions_individual?sites=CSCS-LCG2&sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-11-01&end=2018-11-30&timeRange=daily&granularity=8 Hours&generic=0&sortBy=0&series=All&type=ewa" width="283" /><img alt="" height="211" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/consumptions_individual?sites=CSCS-LCG2&sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-11-01&end=2018-11-30&timeRange=daily&granularity=8 Hours&generic=0&sortBy=0&series=All&type=ewg" width="281" />

 <img alt="" height="181" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/terminatedjobsstatus_individual?sites=CSCS-LCG2&sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-11-01&end=2018-11-30&timeRange=daily&sortBy=0&granularity=8 Hours&generic=0&series=All&type=ebwc" width="290" /><img alt="" height="189" src="http://dashb-atlas-job.cern.ch/dashboard/request.py/efficiency_individual?sites=CSCS-LCG2&sites=UNIBE-LHEP&sitesCat=All Countries&resourcetype=All&projects=All Input Projects&sitesSort=2&sitesCatSort=0&start=2018-11-01&end=2018-11-30&timeRange=daily&granularity=8 Hours&generic=0&sortBy=0&series=All&type=egl" width="252" />
   * *ARC6 upgrade heads-up*
      * At some point in 2019
      * Mayor arc.conf rewrite
      * At the recent NorduGrid developer retreat, I have produced a preliminary conversion to ARC6 of the arc.conf for arc04@lcg.cscs.ch
---+++ UNIBE-ID

 Stable operations in November after the stuck a-rex issue in October<br />During the mid December's maitenance down:Deommissioning of nodes that comprise the el6legacy partition<br />Setup of a subordinate partition for preemptable jobs of the ATLAS experiment at the same tabme

 No sysadmin from UNIBE-ID can join this afternoon

---+++ UNIGE

  Discussing this week how to revive the ARC CE @UniGe

---+++ NGI_CH

 NGI-CH Open Tickets review<br /><br />

| *Ticket-ID* | *Type* | *VO* | *Site* | *Priority* | *Resp. Unit* | *Status* | *Last Update* | *Subject* | *Scope* |
|  <a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=138592" target="_blank">138592 </a>  | | cms | CSCS-LCG2 | urgent | NGI_CH | waiting for reply | 2018-12-06 | Transfers failing from T2_CH_CSCS to ...  | WLCG |
|  <a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=138296" target="_blank">138296 </a>  | | cms | CSCS-LCG2 | urgent | NGI_CH | waiting for reply | 2018-12-05 | Transfers failing from T2_CH_CSCS  | WLCG |
|  <a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=133695" target="_blank">133695 </a>  |  <img alt="" border="0" height="10" src="https://ggus.eu/index.php?mode=download&img=team_ticket.gif" width="38" />  | lhcb | CSCS-LCG2 | urgent | NGI_CH <img alt="" border="0" src="https://ggus.eu/index.php?mode=download&img=tri.gif" /> assigned | waiting for reply | 2018-11-30 | Data access problem at CSCS-LCG2  | WLCG |
|  <a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=131965" target="_blank">131965 </a>  | | none | UNIBE-LHEP | less urgent | NGI_CH <img alt="" border="0" src="https://ggus.eu/index.php?mode=download&img=tri.gif" /> assigned | on hold | 2018-11-15 | IPv6 deployment at WLCG Tier-2 sites  | EGI |
|  <a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=131948" target="_blank">131948 </a>  | | none | CSCS-LCG2 | less urgent | NGI_CH <img alt="" border="0" src="https://ggus.eu/index.php?mode=download&img=tri.gif" /> assigned | in progress | 2018-12-03 | IPv6 deployment at WLCG Tier-2 sites  | EGI |
---++ Other topics

 *Update on experiment share re-balance*
   * Discussed within CHIPP
   * Internal meeting next week to finalise decision
   * Current direction (not final):
      * Reduce max WC for all VOs at the same level
      * Pack single core jobs to nodes (as opposed to spread them)
   * Trial period of 1-2 months<br /><br />Attachment below: ATLAS pending jobs (last 90 days) <p> </p> <p> </p>

<b>Topic2</b>

 Next meeting date:

---++ A.O.B.

---++ Attendants

 CSCS:<br />CMS:<br />ATLAS:<br />LHCb:<br />EGI:

---++ Action items

 Item1<br />ssd-cache-perf.png: <br /><img alt="ssd-cache-perf.png" height="1011" src="%ATTACHURLPATH%/ssd-cache-perf.png" width="1914" />
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	job-accounting-historical-data.png	r1	manage	49.4 K	2018-12-06 - 12:58	GianfrancoSciacca	ATLAS pending jobs (90 days)
png	ssd-cache-perf.png	r1	manage	165.0 K	2018-12-06 - 09:28	DarioPetrusic