MeetingCHIPPCSCSFaceToFace20160901 < LCGTier2

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup,Main.ChippComputingBoardGroup
-->

---+ CHIPP + CSCS Face to Face Meeting on 2016-09-01
   * *Date and time*: Thursday 1st of September at 10:00
   * *Place*: CERN (40-R-B10)
   * *External link / EVO*: No

---++ Agenda
   * 10:00 - Coffee, presentation and agenda
   * 10:15 - VO Status report
      * *LHCb* (30') _Roland absent_ 
      * *ATLAS* (30')
      * *CMS* (30') - Period Jun/Jul/Aug
      * [[http://dashb-cms-jobsmry.cern.ch/tmp/t0AVuWy.png][Wallclock efficiency]] , [[http://dashb-cms-jobsmry.cern.ch/tmp/JeC5op0.png][Failed/Succesful jobs]] , [[http://dashb-cms-jobsmry.cern.ch/dashboard/tmp/20160901.65300.png][Stacked sums]]
      * [[http://dashb-cms-jobsmry.cern.ch/tmp/i6HmAQ7l7.png][Failures split by CMS and Site]] , [[http://dashb-cms-jobsmry.cern.ch/tmp/Tcf7vdVQyBaQ.png][Same as a Pie]]
      * [[http://dashb-cms-jobsmry.cern.ch/tmp/nrLx6nF.png][Pending]] , [[http://dashb-cms-jobsmry.cern.ch/tmp/SsnGK7g.png][Completed]] Jobs by kind of Job - *CMS didn't use CSCS in Aug?*
      * [[https://ggus.eu/?mode=ticket_search&show_columns_check%5B%5D=TICKET_TYPE&show_columns_check%5B%5D=AFFECTED_VO&show_columns_check%5B%5D=AFFECTED_SITE&show_columns_check%5B%5D=PRIORITY&show_columns_check%5B%5D=RESPONSIBLE_UNIT&show_columns_check%5B%5D=STATUS&show_columns_check%5B%5D=DATE_OF_CHANGE&show_columns_check%5B%5D=SHORT_DESCRIPTION&ticket_id=&supportunit=&su_hierarchy=0&former_su=&vo=cms&user=&keyword=cscs&involvedsupporter=&assignedto=&affectedsite=&specattrib=none&status=&priority=&typeofproblem=all&ticket_category=all&mouarea=&date_type=creation+date&tf_radio=1&timeframe=lastquarter&from_date=01+Sep+2016&to_date=02+Sep+2016&untouched_date=&orderticketsby=REQUEST_ID&orderhow=desc&search_submit=GO%21][CMS Tickets in the period]]
   * 11:45 - Tier-2 status and plans & pledges 
      * CSCS (60')
      * UNIBE-LHEP (30')
   * 13:00 - Lunch break
   * 14:00 - Tier-3 status and plans 
      * *PSI* (15')
      * CPUs [[CmsTier3/Tier3Overview#Phase_F_CPU][HW]] , [[http://t3mon.psi.ch/ganglia/PSIT3-custom/running-year.gif][1y Jobs ran]] , [[http://t3mon.psi.ch/ganglia/PSIT3-custom/waiting-year.gif][1y Jobs queued]]
      * Grid Storage [[https://wiki.chipp.ch/twiki/bin/view/CmsTier3/Tier3Overview#Phase_D_Storage][HW]] , [[http://t3mon.psi.ch/ganglia/PSIT3-custom/sespace-year.gif][1y usage]]
      * Shared Cluster Home [[http://t3mon.psi.ch/ganglia/PSIT3-custom/space.report][by ZFS on Linux + NFSv4]]
      * New HW coming in Sep
      * 3 Dalco UIs 
         * each featuring [ 128GB RAM, 2 * E5-2697v4 CPUs, 6*1.8TB 10k disks in *mdadm 1+0*, 2*10GbE ]
      * 1 Grid Storage, type [[http://www.netapp.com/us/products/storage-systems/e2700/e2700-tech-specs.aspx][NetApp E2760]] 
         * [ 52*6TB disks + 8*400GB SSD ], 2 RAID controller SAS based , Net capacity after RAID ~200TB
         * [[https://www.netapp.com/us/media/ds-3395.pdf][NetApp SANtricity SSD Cache]] , [[https://www.netapp.com/us/media/ds-3309.pdf][NetApp SANtricity Dynamic Disk Pools]]
      * UNIBE-ID (?)
      * UNIGE (15'): [[https://wiki.chipp.ch/twiki/pub/LCGTier2/MeetingCHIPPCSCSFaceToFace20160901/Grid_FTF_2016_Sept_1.pdf][Slides]]
   * 14:45 - NGI_CH Gianfranco (45')
   * 15:30 - Open discussion (60')
   * 16:30 - A.O.B. / End of meeting

---++ Attendants
   * Christoph, Derek, Gianfranco, Fabio, Luis, Dino, Dario, Gianni, Miguel, Stefano, Pablo

---++ Minutes and action items
   * Attached documents show the information presented during the meeting. Besides, it was agreed that:
   * ATLAS currently sees a big duplication of efforts (GF has to explain the issue, CSCS has to understand it). If those were the same person that would be more efficient.
   * (action on %RED%Gianfranco%ENDCOLOR%) ATLAS should communicate the average efficiency of all ATLAS sites during the whole period, so as to understand where some of those 'dips' come from
   * (action on *CSCS*) CSCS should try to explain where some of those efficiency dips come from during the last year, so as to understand if there is a good explanation (such as known incidents) or not.
   * Gianfranco's dedication to ATLAS exclusively amounts to an average of 0.4 FTE during the last 4 years
   * In the "options to move forward" page, for case A (continue as we are, with improvements) manpower at the site needs a good dashboard and good connection with the VO to understand the logs, know how to compare the site with others, and better ability to move around VO internals.
   * (action on *ALL*) it was agreed that both the VO Reps and CSCS would sit down together in a 2-day monitoring hackathon to improve on the discovery and response of issues by developing a dashboard with CSCS and all VO's view of important metrics. The availability window is not opened until the second half of October.
   * (action on *ALL*) we need to add the main sysadmins @ CSCS to all VOs. %BLUE%Fabio%ENDCOLOR% : sent howto %Y%
   * (action on %BLUE%Fabio%ENDCOLOR% and *CSCS*) it would be interesting for CSCS to read the minutes from the CMS weekly operations call (Fabio will attend and send the minutes). [[https://twiki.cern.ch/twiki/bin/view/CMS/FacilitiesServicesMinutes][Minutes Archive]]. %Y%
   * (action on %BLUE%Fabio%ENDCOLOR%) CSCS mailing list =grid-list@cscs.ch= needs to receive the CMS Nagios warnings. %Y%
   * %BLUE%Fabio%ENDCOLOR% is doing the first level support for CMS, but having CSCS look into it should improve not only CMS but the rest of the VOs. [[CMSMonitoring][Fabio's T2 checks]] %Y%
   * CMS does not see the efficiency problems that ATLAS reports. 
   * (action on %BLUE%Fabio%ENDCOLOR% and *CSCS*) we need to match the CMS and ATLAS efficiencies to see where they match (probably a site-wide problem) and where they differ (probably a VO-specific problem)
   * (action on *ALL*) Official Availability/Reliability metrics from WLCG is not real. For next time, we need to agree on a A/R figure that represents reality.
   * (action on *CSCS*) per-VO CPU accounting data should be added to the plot
   * VO-Reps can now login to servers/nodes from =login.lcg.cscs.ch= (from =ela.cscs.ch=) as their own user to do certain operations with sudo (such as =cat /var/log/*= and such). If they see a need for more operations they should open a ticket
   * (action on %ORANGE%Derek%ENDCOLOR%) we should expect an answer for the VO-Box proposal before end of September
   * (action on *CSCS*) it was agreed that we should have a bi-weekly call to review new and old tickets. All issues have to be on the table and shared with everyone (ticket or not). 
   * (action on %RED%Gianfranco%ENDCOLOR% and *CSCS*) we need to look into the ARC configuration (and the VO queue config) to see if it is correct, since there is a more-than-10x increase in the usage of the scratch file system in phoenix since April'16
   * (action on *CSCS*) we need to understand the impact of not imposing memory limits (e.g. do nodes swap?)
   * (action on *CSCS*) authentication on Kibana (for graphs/logs sharing) should be done with Grid certificates


---++ Other

   * [[%ATTACHURL%/CSCS-ATLASreport-20160901.pdf][CSCS-ATLASreport-20160901.pdf]]: CSCS-ATLASreport

   * [[%ATTACHURL%/UNIBE-LHEP-20160901.pdf][UNIBE-LHEP-20160901.pdf]]: UNIBE-LHEP T2 site report
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pptx	20160901_F2F_Stefano.pptx	r1	manage	2303.8 K	2016-09-01 - 10:26	StefanoGorini
pdf	CSCS-ATLASreport-20160901.pdf	r1	manage	1879.4 K	2016-09-01 - 07:42	GianfrancoSciacca	CSCS-ATLASreport
pdf	Grid_FTF_2016_Sept_1.pdf	r1	manage	3887.4 K	2016-09-01 - 07:56	LuisMarch	UNIGE Tier-3 ATLAS Cluster
pdf	UNIBE-LHEP-20160901.pdf	r1	manage	1859.9 K	2016-09-01 - 10:14	GianfrancoSciacca	UNIBE-LHEP T2 site report
This topic: LCGTier2 > WebHome > MeetingsBoard > MeetingCHIPPCSCSFaceToFace20160901
Topic revision: r20 - 2016-09-07 - FabioMartinelli