CHIPP + CSCS Face to Face Meeting on 2016-09-01

• Date and time: Thursday 1st of September at 10:00
• Place: CERN (40-R-B10)
• External link / EVO: No

Agenda

• 10:00 - Coffee, presentation and agenda
• 10:15 - VO Status report
  • LHCb (30') Roland absent
  • ATLAS (30')
  • CMS (30') - Period Jun/Jul/Aug
  • Wallclock efficiency , Failed/Succesful jobs , Stacked sums
  • Failures split by CMS and Site , Same as a Pie
  • Pending , Completed Jobs by kind of Job - CMS didn't use CSCS in Aug?
  • CMS Tickets in the period
• 11:45 - Tier-2 status and plans & pledges
  • CSCS (60')
  • UNIBE-LHEP (30')
• 13:00 - Lunch break
• 14:00 - Tier-3 status and plans
  • PSI (15')
  • CPUs HW , 1y Jobs ran , 1y Jobs queued
  • Grid Storage HW , 1y usage
  • Shared Cluster Home by ZFS on Linux + NFSv4
  • New HW coming in Sep
  • 3 Dalco UIs
    • each featuring [ 128GB RAM, 2 * E5-2697v4 CPUs, 6*1.8TB 10k disks in mdadm 1+0, 2*10GbE ]
  • 1 Grid Storage, type NetApp E2760
    • [ 52*6TB disks + 8*400GB SSD ], 2 RAID controller SAS based , Net capacity after RAID ~200TB
    • NetApp SANtricity SSD Cache , NetApp SANtricity Dynamic Disk Pools
  • UNIBE-ID (?)
  • UNIGE (15'): Slides
• 14:45 - NGI_CH Gianfranco (45')
• 15:30 - Open discussion (60')
• 16:30 - A.O.B. / End of meeting

Attendants

• Christoph, Derek, Gianfranco, Fabio, Luis, Dino, Dario, Gianni, Miguel, Stefano, Pablo

Minutes and action items

• Attached documents show the information presented during the meeting. Besides, it was agreed that:
• ATLAS currently sees a big duplication of efforts (GF has to explain the issue, CSCS has to understand it). If those were the same person that would be more efficient.
• (action on Gianfranco) ATLAS should communicate the average efficiency of all ATLAS sites during the whole period, so as to understand where some of those 'dips' come from
• (action on CSCS) CSCS should try to explain where some of those efficiency dips come from during the last year, so as to understand if there is a good explanation (such as known incidents) or not.
• Gianfranco's dedication to ATLAS exclusively amounts to an average of 0.4 FTE during the last 4 years
• In the "options to move forward" page, for case A (continue as we are, with improvements) manpower at the site needs a good dashboard and good connection with the VO to understand the logs, know how to compare the site with others, and better ability to move around VO internals.
• (action on ALL) it was agreed that both the VO Reps and CSCS would sit down together in a 2-day monitoring hackathon to improve on the discovery and response of issues by developing a dashboard with CSCS and all VO's view of important metrics. The availability window is not opened until the second half of October.
• (action on ALL) we need to add the main sysadmins @ CSCS to all VOs. Fabio : sent howto
• (action on Fabio and CSCS) it would be interesting for CSCS to read the minutes from the CMS weekly operations call (Fabio will attend and send the minutes). Minutes Archive.
• (action on Fabio) CSCS mailing list grid-list@cscs.ch needs to receive the CMS Nagios warnings.
• Fabio is doing the first level support for CMS, but having CSCS look into it should improve not only CMS but the rest of the VOs. Fabio's T2 checks
• CMS does not see the efficiency problems that ATLAS reports.
• (action on Fabio and CSCS) we need to match the CMS and ATLAS efficiencies to see where they match (probably a site-wide problem) and where they differ (probably a VO-specific problem)
• (action on ALL) Official Availability/Reliability metrics from WLCG is not real. For next time, we need to agree on a A/R figure that represents reality.
• (action on CSCS) per-VO CPU accounting data should be added to the plot
• VO-Reps can now login to servers/nodes from login.lcg.cscs.ch (from ela.cscs.ch) as their own user to do certain operations with sudo (such as cat /var/log/* and such). If they see a need for more operations they should open a ticket
• (action on Derek) we should expect an answer for the VO-Box proposal before end of September
• (action on CSCS) it was agreed that we should have a bi-weekly call to review new and old tickets. All issues have to be on the table and shared with everyone (ticket or not).
• (action on Gianfranco and CSCS) we need to look into the ARC configuration (and the VO queue config) to see if it is correct, since there is a more-than-10x increase in the usage of the scratch file system in phoenix since April'16
• (action on CSCS) we need to understand the impact of not imposing memory limits (e.g. do nodes swap?)
• (action on CSCS) authentication on Kibana (for graphs/logs sharing) should be done with Grid certificates

Other

• CSCS-ATLASreport-20160901.pdf: CSCS-ATLASreport

• UNIBE-LHEP-20160901.pdf: UNIBE-LHEP T2 site report