April-20

• Slurm
  • memory is configured as consumable resource (default DefMemPerCPU is 2GB/CPU) to prevent out of memory situations caused by users jobs
  • added to client nodes LNAG enviromental variables to /etc/locale.conf to shut out LC_CTYPE/UTF-8 errors of ssh-sessions

• Monitoring:
  • added Slurm CPU/GPU metric collection scripts and plots: https://wiki.chipp.ch/twiki/bin/view/CmsTier3/SlurmUtilisation
  • added Admins monitoring list: https://wiki.chipp.ch/twiki/bin/view/CmsTier3/MonitoringList

• Miscellaneous:
  • CRIC/SRR storage monitoring ticket closed: storage descriptor is configured on t3dcachedb03
  • updates of EGI Trust Anchor release 1.105-1
  • users question to install phython3/root6 locally not needed, since availble in /cvmfs/sft.cern.ch/lcg/...
  • migration of puppet filecopy location to common for all t3admins gitlab place
  • user accounts/data cleaning (jfernan2, thaarres), creating of new UniZ accounts (sliechti, yverma)

March-20

• dCache Upgrade Follow-ups:
  • add CMS TFC config to xrootd door on SE node (https://www.dcache.org/downloads/xrootd4j/index.shtml)
  • implementation of Postgres Backup script to copy DB to t3nfs02:/zfs/data01/swshare/postgres
  • dcache after upgrade became too verbose and filled out /var/log partition; to fix the problem dcache restart was done on Sun Mar 15 (without user activity)
• Storage Cleaning due to almost no free space on dcache:
  • deletion of leftover user data took several days. Too many (hundred thousands) files in single directories: dcache can't handle it
  • overal clenup brought ~ 30% free space; next step is needed - check and clean ~150TB of mc, data dirs
• Slurm:
  • add QoS (500 cpu/user) to quick partition
  • EOS test configuration (enabled on Worker Nodes and UIs): since February no user feedback
• Monitoring:
  • manually added non-standard /work server t3nfs02 to ganglia
  • solved the problem with SELinux (the reason of http access error) on ganglia server; works stably
  • dcache space monitoring added to t3wiki: https://wiki.chipp.ch/twiki/bin/view/CmsTier3/StoragePlots
• all configuration changes saved on hiera/puppet/gitlab
• most of this list was done remotely from home with no drop in efficiency in compare to work from PSI office