(r9) MeetingSwissGridOperations20200220 < LCGTier2

Tags: view all tags
&lt;!-- keep this as a security measure:<br /> * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.EgiGroup<br /> * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup<br /> #uncomment this if you want the page only be viewable by the internal people<br /> #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.ChippComputingBoardGroup<br />--&gt;

---+ Swiss Grid Operations Meeting on 2020-02-20 at 15:30
   * *###############################*
   * ## CSCS Meeting room 1st floor (LCA) <br /><br />Join by Zoom meeting:<br /> https://cscs.zoom.us/my/cscsmr1f <br />Host key: 656565<br /><br />Join by phone:<br />Switzerland: +41 43 210 70 42 | Meeting ID 9161083650<br />International phone access numbers: https://zoom.us/u/amdNZ9YRF <br /><br />Join by SIP:<br /> [[mailto:9161083650@zoomcrc.com][9161083650@zoomcrc.com]]
   * *###############################*
   * ## For people at CERN you can come to my office 32 - 4 C16

%TOC%
---++ Followup from previous Action Items

[[MeetingT2DiscussionETH20200116#Action_items_9][Action items]]
---++ VO reports
---+++ ATLAS
   * [[%ATTACHURL%/ATLAST2reportJan2020pdf.pdf][ATLAST2reportJan2020pdf.pdf]]: ATLAS CH Tier2 report
Minutes:
   * the pledges were achieved (surpassed) in the past month
   * while the pledged performance were achieved, shortfall on the 40:40:20 share / see Nick slides to understand why the 40:40:20 was difficult to achieve last month
   * 15-20% of the 1-core EVGEN jobs timeout w/o logs. Debugging with Miguel not cocnlusive yet. Impossible to save the session for all jobs from ATLAS--&gt; try to do it for single core production jobs --&gt; then debug
   * split the monitoring plot of "Slots of running jobs" in two plots - just not to have the stacked plots. The total pledged is automatically added in the plots, add by hand a dotted line with the CSCS-only pledge
   * All dips in the plots for both CSCS and Bern discussed and understood
   * The color coding in the table "ATLAS T2-statistics" is red &lt; 75% &lt; yellow &lt; 90%&lt; green
   * The color coding in the table "ATLAS Hammercloud statistics" is red &lt; 95% &lt; green
   * Some details about the Swiss ATLAS federation will be send in the next days
---+++ CMS

Minutes:
   * a ticket has been filed to understand why CMS had so few pending jobs (the initial explanation is that there were no jobs to run in the whole CMS which sounds surprising)
---+++ LHCb

Minutes:
   * LHCb all OK
   * 2 misconfigurations occurred on the LHCb side
   * affected by the SEF(?) downtime
   * IP6 problem solved
---++ T2 Sites reports
---+++ CSCS
   * [[%ATTACHURL%/CHIPPreportJan2020.pdf][CSCS January Site Report]]
January utilization:
   * Pledges
      * 112.2% CHIPP overall
      * 104.1% ATLAS
      * 101.5% CMS
      * 149.9% LHCb
   * Sharing
      * ATLAS:CMS:LHCb [%] 37:36:27
Minutes:
   * discussion of the dried out queues
   * Whenever a VO does not submit jobs (e.g. by watching the grafana page) warn CSCS and try to debug the case as much in real time as possible
   * ATLAS has a procedure to drain its queue for scheduled downtimes. This explains why the number of jobs went down before the actual maintenance. After the maintenance the jobs where back in a few ours as expected. While this is not necessarily a problem per se, the mechanism has an impact on the capability to reach the pledges.
   * We badly need to improve the communnication flow between VOs and CSCS. Try with:
      * Hot topics page (see <a href="HotTopics" target="_blank">Hot topics page</a>) to collect changes to the system that can possibly affect operations
      * Slack-chat in case special "real time" debugging is needed while the issue is in progress
      * tickets to flag problems
---+++ UNIBE LHEP/Ubelix
   * [[%ATTACHURL%/UnibeT2ReportJan2020.pdf][UnibeT2ReportJan2020.pdf]]: UniBE Tier2 report
<span style="background-color: transparent;">Minutes:</span>
   * no particular discussion
---++ T3 Sites reports
---+++ PSI
   * Conitinue migration of all T3 nodes to rhel7: Slurm clients were done
   * T3 downtime day for Storage Upgrade. Due to good preparation and testing the following upgrades were completed:
      * dCache servers OS from sl6 to rhel7
      * dCache from 3.2 to 5.2
      * Postgresql from 9.5 to 11
      * Firmware on Dalco storage pools and !NetApp

   * There was storage discussion at T3 regarding POSIX perspective solutions to replace egeing hardware (40TB ZFS on Linux). So that here is my review summary concerning usage of dCache as NFS4.1 server: still looks like a bleeding edge development, needs constantly to update dCache and configurations.
      * no user quota
      * files are immutable
      * experience report from NIKHEF (very few sites running nfs4.1/pnfs with dcache): https://indico.desy.de/indico/event/22170/session/1/contribution/6/material/slides/1.pdf
      * recent comments from NIKHEF: - might stuck on client side with intensive IO - recommendation to update to vers. 6 (not yet Golden release)

---+++ UniGE
   * Can now request grid certificates via the Swiss CA
   * DPM pools upgraded to the latest version in line with the Bern ones
   * ARC CE deployment delayed, will have a meeting next Monday to outline the final steps
---++ EGI
---+++ News
   * NTR
---+++ Review of open tickets
   * https://ggus.eu/index.php?mode=ticket_search&supportunit=NGI_CH&status=open&timeframe=any&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO<br /><br /> <table border="0" cellpadding="2" cellspacing="0" style="width: 1109px; height: 384px;"> <tbody> <tr> <td colspan="5"> *8 Tickets found* </td> </tr> <tr> <td> *Ticket-ID* </td> <td> *Type* </td> <td> *VO* </td> <td> *Site* </td> <td> *Priority* </td> <td> *Resp. Unit* </td> <td> *Status* </td> <td> *Last Update* </td> <td> *Subject* </td> <td> *Scope* </td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=145582" target="_blank">145582 </a></td> <td align="center"> </td> <td>cms</td> <td>CSCS-LCG2</td> <td>urgent</td> <td>NGI_CH</td> <td>in progress</td> <td>2020-02-19</td> <td align="left" width="250px">T2_CH_CSCS is intermittently failing ...</td> <td>WLCG</td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=144898" target="_blank">144898 </a></td> <td align="center"> </td> <td>cms</td> <td>CSCS-LCG2</td> <td>less urgent</td> <td>NGI_CH</td> <td>in progress</td> <td>2020-01-22</td> <td align="left" width="250px">T2_CH_CSCS warning - outdated version ...</td> <td>WLCG</td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=144499" target="_blank">144499 </a></td> <td align="center"> </td> <td>none</td> <td>T3_CH_PSI</td> <td>less urgent</td> <td>NGI_CH</td> <td>assigned</td> <td>2020-02-18</td> <td align="left" width="250px">Upgrade to recent dCache release</td> <td>EGI</td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=144485" target="_blank">144485 </a></td> <td align="center"> </td> <td>none</td> <td>CSCS-LCG2</td> <td>less urgent</td> <td>NGI_CH</td> <td>assigned</td> <td>2020-02-04</td> <td align="left" width="250px">Upgrade to recent dCache release</td> <td>EGI</td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=143464" target="_blank">143464 </a></td> <td align="center"> </td> <td>none</td> <td>UNIBE-LHEP</td> <td>urgent</td> <td>NGI_CH</td> <td>in progress</td> <td>2020-02-20</td> <td align="left" width="250px">DPM at UNIBE-LHEP has to be configured ...</td> <td>EGI</td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=141276" target="_blank">141276 </a></td> <td align="center"> </td> <td>none</td> <td> </td> <td>less urgent</td> <td>NGI_CH <img alt="" border="0" src="https://ggus.eu/index.php?mode=download&img=tri.gif" /> assigned</td> <td>on hold</td> <td>2019-11-26</td> <td align="left" width="250px">yearly review of the information ...</td> <td>EGI</td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=131965" target="_blank">131965 </a></td> <td align="center"> </td> <td>none</td> <td>UNIBE-LHEP</td> <td>less urgent</td> <td>NGI_CH <img alt="" border="0" src="https://ggus.eu/index.php?mode=download&img=tri.gif" /> assigned</td> <td>on hold</td> <td>2020-01-20</td> <td align="left" width="250px">IPv6 deployment at WLCG Tier-2 sites</td> <td>EGI</td> </tr> <tr> <td align="center" width="60px"><a href="https://ggus.eu/index.php?mode=ticket_info&ticket_id=131432" target="_blank">131432 </a></td> <td align="center"> </td> <td>none</td> <td>CSCS-LCG2</td> <td>urgent</td> <td>NGI_CH <img alt="" border="0" src="https://ggus.eu/index.php?mode=download&img=tri.gif" /> assigned <img alt="" border="0" src="https://ggus.eu/index.php?mode=download&img=tri.gif" /> involved</td> <td>in progress</td> <td>2020-01-27</td> <td align="left" width="250px">Storage accounting deployment</td> <td>EGI</td> </tr> </tbody> </table> <p> </p>
---++ a.o.b
   * Summary of the discussion at CSCS on 10.02.2020 [[%ATTACHURL%/20200220_OpsMeeting.pdf][20200220_OpsMeeting.pdf]]
   * Next meeting date: 05.03.2020 --&gt; ATLAS GPU challenge + discussion of the slides above
---++ Attendants
   * CSCS: Nick, Pablo, Dino, Dario
   * CMS: Mauro, Christoph, Derek
   * ATLAS: Gianfranco
   * LHCb:Roland
   * EGI: Gianfranco
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	20200220_OpsMeeting.pdf	r1	manage	47.3 K	2020-02-20 - 12:36	MauroDonega
pdf	ATLAST2reportJan2020pdf.pdf	r1	manage	403.0 K	2020-02-20 - 13:57	GianfrancoSciacca	ATLAS CH Tier2 report
pdf	CHIPPreportJan2020.pdf	r2 r1	manage	5618.4 K	2020-02-19 - 07:03	NickCardo	CSCS January Site Report
pdf	UnibeT2ReportJan2020.pdf	r1	manage	186.7 K	2020-02-20 - 13:58	GianfrancoSciacca	UniBE Tier2 report