Announcing a Tier-3 shutdown

• Announce the system halt on the GOC Pages (24h before if it is a scheduled downtime)
• Announce the system halt on cms-tier3-users@lists.psi.ch

Shutting down the Tier-3

1. Prevent further logins to the user interfaces. Modify /etc/security/access.conf on the user interfaces by commenting out the lines that allow access for all CMS users and for NX

   #+ : cms : :0 129.129.0.0/16
   + : feichtinger loktionova_n : : ALL
   #+ : cms : ALL
   #+ : nx : ALL
   - : ALL : ALL
   
      

2. ssh root@t3nagios /etc/init.d/nagios stop
3. disable queues on WNs: ssh t3ce02 qmod -d @ and kill rest of jobs
4. t3ui* /etc/nologin
5. 4.1 umount /pnfs an all nodes: UIs, WNs, t3se01 and t3dcachedb03 4.2 comment in fstab /pnfs line to prevent mount after reboots
6. service xrootd/cmsd stop on t3se01
7. dcache stop 4.1 doors (xrootd, dcap/gsidcap, gsifttp, srm, xrootd) 4.2 pools: t2fs07-10, t3nfs02 4.4 se + db
8. stop zookeeper on t3zkpr01-03
9. phedex, bdii, frontier - VMs ?
10. shutdown t3nfs01, t3nfs02, t3fs07-11, t3gpu01, t3admin01/02
11. Shut down Netapp system (Link)
    • Make sure, no background processes in operation (Santricity SMclient GUI)
    • Turn off controller enclosure
    • (Turn off any additional enclosure)

-- DerekFeichtinger - 2019-01-03