Swiss Grid Operations Meeting on 2015-10-15

• Date and time: First Thursday of the month, at 14:00
• Place: Vidyo (room: Swiss_Grid_Operations_Meeting, extension: 109305236)
• External link: http://vidyoportal.cern.ch/flex.html?roomdirect.html&key=gDf6l4RlIAGN
• Phone gate: From Switzerland: 0227671400 (portal) + 109305236 (extension) + # (pound sign)
• IRC chat: irc:gridchat.cscs.ch:994#lcg (ask pw via email)

Site status

CSCS

• dCache
  • Security upgrade to version 2.6.52 also planned to upgrade to last version next month, Dario will perform the upgrade.
  • Dario distributed the remaining storage according to our last f2f meeting.
  • We set up a downtime for tomorrow October 16 to attempt to resolve dCache issues experienced this week
• Network
  • Infiniband switches and bridges upgraded to last version.

PSI

• dCache
  • upgraded from 2.10 to the latest 2.13.9 ; this was an easy upgrade, CSCS might try the upgrade 2.6 -> 2.10 -> 2.13 in order to avoid a downtime
  • upgraded accordingly the xrootd monitoring plugin RPM and its conf
  • upgraded to the latest Xrootd RPMs 4.2.3* More... Close

    cms-xrootd-dcache-1.2-7.osg.el6.noarch gfal2-plugin-xrootd-0.3.4-1.el6.x86_64 xrootd-4.2.3-1.el6.x86_64 xrootd-client-4.2.3-1.el6.x86_64 xrootd-client-libs-4.2.3-1.el6.x86_64 xrootd-cmstfc-1.5.1-10.osg32.el6.x86_64 xrootd-fuse-4.2.3-1.el6.x86_64 xrootd-libs-4.2.3-1.el6.x86_64 xrootd-selinux-4.2.3-1.el6.noarch xrootd-server-4.2.3-1.el6.x86_64 xrootd-server-libs-4.2.3-1.el6.x86_64 

  • my materialized views still work 'as they are' with 2.13.9
  • instead the Derek's tools need to be updated because of the new 2.13 Admin door commands ( e.g. not cd Cell but \c Cell ) ; I've partially updated them by a bit of sed but it's still a working in progress.
• Latest Python packages on SL6
  • This is mainly addressed to the T3s because they face the final user issues ; at PSI Scientists use Anaconda on SL6 in order to easily use an updated and extended Python distribution ; I run an installation of Anaconda as well, it's very easy both use it and update it
• NFSv4
  • In order to replace our very old NFSv3 services based on Solaris 10 we've eventually ordered :
  • HP G9 DL380 24SFF wih 24*600GB SAS 15k 2.5" + 2*146GB SAS 15k 2.5"
  • HP G9 DL380 12LFF wih 12*3TB SATA 7.2k 3.5" + 2*146GB SAS 15k 2.5"
  • the former will act as a NFSv4 service, the latter as a backup server ; backups by the ZFS send/receive mechanism.
  • servers RAID controller will run in HBA mode because of ZOL
  • servers will run CentOS7 and ZOL
  • CentOS7 hardening by CIS RHEL7
  • wanna make my 1st PXE UEFI installation ; after several unsuccessful trials I've created this grub.cfg and kickstart file
  • I also need mdadm RAID1 by kickstart over the 2*146GB OS disks
  • Client side I'm planning to use FS-Cache
• Nagios4
  • Upgraded to Nagios 4.1.1
• Compact UIs/WNs featuring many disks
  • As the # of CPUs core grow we've to accordingly scale the # of disks ( and the net/RAM/.. ) ; CSCS uses a distributed GPFS scratch but in respect of our yearly budget we can just select DAS solutions ; in this context I've enquired the market, outcome is :
  • Dalco offers the new 2u Intel chassis with 24 SFF disks, 4 servers, 6 disks per server, a dedicated RAID5 controller per server
  • HP offers the 2u Apollo chassis ; nice but it's ~20% more expensive than Intel
  • If you're willing to run mdadm RAID10 then Dalco offers the older but even cheaper 2u Intel chassis with 16 SFF disks ; because I know how to manage mdadm RAID10 this is my current choice but obviously the dedicated RAID5 controller per server reduces the Sys Admin efforts.
• CMS PhEDEx
  • Updated to PHEDEX_4_1_5
• Son of Grid Engine and cgroups
  • No progress, too busy.

UNIBE-LHEP

• Operations
  • Relatively smooth running period (partly unattended)
  • I/O errors on lustre mds (ce02) due to a degraded RAID10. Power-cycled, lustre self-recovered relatively quickly
  • One more cooling incident last week: power loss to many wn's, lustre mds needed power-cycling (lustre self-recovered within ~1h)
• ATLAS specific operations
  • Nothing specific to report
  • ATLAS have made a hufe progress on task definitions and makinf the workflows considerably more failsafe, making sites' life considerably simpler
  • Also many ARC bugfixes have contributed
  • Delivery vs pledges: http://dashb-atlas-job.cern.ch/dashboard/request.py/resourceutilization_individual?sites=CSCS-LCG2&sites=UNIBE-LHEP&sites=UNIGE-DPNC&sitesCat=CH-CHIPP-CSCS&resourcetype=All&activities=analysis&activities=analy_test&activities=dbgmerge&activities=dbgrec&activities=digit&activities=eiupload&activities=eventindex&activities=evgen&activities=gangarobot&activities=gangarobot-mcore&activities=gangarobot-new&activities=gangarobot-nightly&activities=gangarobot-pft&activities=gangarobot-rctest&activities=hammercloud&activities=idalignmerge&activities=idalignreco&activities=idalignsolve&activities=install&activities=larntupmerge&activities=larrecon&activities=merge&activities=pile&activities=pmerge&activities=ptest&activities=rc_test&activities=reco&activities=recon&activities=reprocessing&activities=sctbse&activities=sctdc&activities=sctds&activities=scteff&activities=scthm&activities=sctla&activities=sctno&activities=sctns&activities=sctro&activities=select&activities=simul&activities=skim&activities=t0_aodmerge&activities=t0_hist1merge&activities=t0_histmerge&activities=t0_ntupmerge&activities=t0_rawmerge&activities=t0_recon&activities=t0_tagmerge&activities=test&activities=trt1merge&activities=trt2merge&activities=trtaccu&activities=trtcalib&activities=trtcalmerge&activities=validation&sitesSort=7&sitesCatSort=2&start=2015-04-01&end=2015-10-15&timeRange=daily&granularity=Daily&generic=0&sortBy=0&diag1=0&diag2=0&diag3=0&diag4=0&diag5=0&diag6=0&diag7=0&diag8=0&diagT=0&diag8pl=0&series=30&type=cpuhs
  • 
    
  • Cluster re-installation workflow development finalised:
    • Rocks 6.2
    • SLC 6.7
    • ARC 5.0.3
    • SLURM 15.08.0-1
    • Lustre 2.5.3
  • Plan to re-build ce01 starting <insert free time here>
  • Temperature monitoring in progress (room/racks/servers) - Very useful input from PSI :)

UNIBE-ID

• Operations
  • Smooth operatiosn, only few minor issues
• Storage Migration
  • Migration plan developed to move from generic GPFS cluster to new IBM ESS
  • Online data migration since 3 weeks (though slow due to compute workload on ethernet)
  • Planned downtime and moving all nodes to the new GPFS cluster on 2015-10-08 (yes, today, therefore no attendee from UNIBE-ID)

UNIGE

• New admin: please welcome Luis March Ruiz

NGI_CH

• Update on Certificates
• ARGO Availabilities site reports: http://argo.egi.eu/lavoisier/site_reports?ngi=NGI_CH
• MyEGI monitoring portal: http://mon.egi.eu/myegi/sa/?view=2&graph=1&vo=104&profile=26&filters-value-Regions_or_Tiers=115&filters-value-Sites=&production=1&preproduction=1&period=pM&dateorperiod=dt&startdate=01-09-2015&enddate=15-10-2015&resolution=D

Other topics

• Update from ATLAS (CSCS):
  • arc02 status
  • decommissioning of PRODDISK
  • dark data in dCache: http://adc-ddm-mon.cern.ch/ddmusr01/T2_DATADISK.html
  • new batch shares: 75% for production, 25% for analysis (was 50/50) - within production: 80% mcore, 20% score
• Topic2

A.O.B.

Attendants

• CSCS: Pablo Fernandez, Dino Conciatore
• CMS: Fabio Martinelli, Daniel Meister (by phone)
• ATLAS: Gianfranco Sciacca, Luis March Ruiz
• LHCb: none
• EGI: Gianfranco Sciacca

Action items

• Item1