(r11) UpgradePlanning201301100953 < CmsTier3

Tags: view all tags
<!-- keep this as a security measure:
   #uncomment if the subject should only be modifiable by the listed groups 
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup
   #uncomment this if you want the page only be viewable by the listed groups
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup
-->

*CMS Tier-3 Upgrade Planning Page*

<!--
  # Use the attached form to define title and summary of this news item. The details you can fill in directly on this wiki page.
-->

---+ %FORMFIELD{"Title"}%

---++ Summary
%FORMFIELD{"Summary"}%

---++ Details

Since some of our Switches will get SW upgrades, we will also lose all T3 intranet connectivity for a short while. The virtualization infrastructure will be offline for some considerable time over this weekend, so a lot of our service nodes will be down.
We should shut down the UIs and WNs.

---+++ Notes

This is the link for the general PSI compute center shutdown on 2013-01-12: https://intranet.psi.ch/AIT/SystemTest2013

   * *t3ce01*
      * What for are the rsync jobs that seem to copy the ganglia rrds continually to the /dev/shm? Seem to be ancient rsync stuck jobs. I deleted them
      * t3ce01 still runs all ganglia connectors
   * *t3mon01*
      * shows *high memory and CPU usage*. Might be an error condition that arose on Fri 11th, but also I see that ganglia is not implemented in the way I had prototyped for CSCS and PSI: The RRD data base files should reside in memory and should get synched to a disk area every few minutes. That prevents the system from being overcome by the mani IOPs that are stupidly generated by so many RRD db files (and the way ganglia is organized, there is one file for every metric and host)
   * Minor issues
      * *t3admin01*
         * please remove obsolete files that are clogging up /root directory
      * Terminal colors on newer nodes: Please disable color output. It is not readable for people who use white background (e.g. yellow color on white is invisible)
 



---+++ Shutting down of all services
_Remarks added by Fabio_
   1. Stop Nagios: =ssh root@t3nagios /etc/init.d/nagios stop=
   1. Prevent users from logging in by modifying =/etc/security/access.conf=. Reboot all UIs to get rid of sessions. <verbatim>
Only t3ui05 and t3ui06 managed to reboot! 
t3ui03 I had to shut off manually, since the ILOM has not been reachable since last summer. Even after manual power on, it did not come back up.
The other machines I had to forcefully shut down and turn on using the ILOMs.

[root@t3admin01 MAINTENANCE]# cexec ui: uptime
************************* ui *************************
--------- t3ui02---------
ssh: connect to host t3ui02 port 22: Connection refused
--------- t3ui03---------
ssh: connect to host t3ui03 port 22: No route to host
--------- t3ui04---------
ssh: connect to host t3ui04 port 22: No route to host
--------- t3ui05---------
 16:09:05 up 3 min,  0 users,  load average: 0.29, 0.23, 0.10
--------- t3ui06---------
 16:09:06 up 3 min,  0 users,  load average: 0.20, 0.20, 0.09
--------- t3ui07---------
ssh: connect to host t3ui07 port 22: No route to host
</verbatim>
   1. Kill all surviving SGE jobs.
   1. Shut down SGE on all WNs: =cexec wn: /etc/init.d/sgeexecd.p6444 stop= and =cexec intel: /etc/init.d/sgeexecd.p6444 stop=
   1. Shut down SGE master: =ssh t3ce "/etc/init.d/sgedbwriter.p6444 stop; /etc/init.d/sgemaster.p6444 stop"= _There are a couple of nohup tail running based on the SGE pid, please leave SGE ON, simply start iptables_ =ssh t3ce service iptables start=
   1. Stop Crond on t3se01,02,t3dcachedb01,04 _t3dcachedb01 cron already OFF since last upgrade, same for NEW t3se02,t3dcachedb02_ 
   1. Shut down phedex services on t3cmsvobox
   1. shut down all worker nodes
   1. <strike>shut down all UIs</strike> I decided to leave the UIs up (Derek)
   1. Stop the production dcache (StartStopDcache26)
   1. *SKIP* Stop NFS swshare services (=showmount -a= for finding clients)
      1. unmount NFS home from all WNs and UIs _remember cexec wn: and cexec intel: , or simply add to wn: also the intel WNs once and forever_
      1. t3cmsvobox02.psi.ch:/swshare
      1. t3vmui01.psi.ch:/swshare
      1. STALE: t3vm01.psi.ch:/swshare
   1. *SKIP* Stop NFS shome services (=showmount -a= for finding clients)
      * all UIs
      * t3vmui01.psi.ch:/shome
      * all WNs
      * t3ce01.psi.ch:/shome
      * t3ce02.psi.ch:/shome
      * t3dcachedb01.psi.ch:/shome
      * t3dcachedb02.psi.ch:/shome/monuser
      * t3dcachedb04.psi.ch:/shome/monuser
      * t3ldap01.psi.ch:/shome
      * t3mon01.psi.ch:/shome
      * t3nagios.psi.ch:/shome/martinelli_f
      * t3nfs01.psi.ch:/shome _outdated_
      * t3se02.psi.ch:/shome/martinelli_f _outdated_
      * t3cmsvobox02.psi.ch:/shome
      * DEAD ---t3vm01.psi.ch:/shome---
      * DOWN ---t3dcachedb03.psi.ch:/shome---
      * DEAD ---t3vm03.psi.ch:/shome---
      * DEAD ---t3vmmaster01.psi.ch:/vmshare---



---+++ dCache Upgrade
Assuming both =t3se01= and =t3dachedb04= UP but dCache stopped everywhere ( also on =t3se02= and =t3dcachedb02= ):
   1. Stop =t3se01= because =t3se02= will become the new =t3se01=
   1. Stop =t3dcachedb04= Crond if found ON.
   1. Unfortunately all our daily =pgdumpall= backups are probably not consistent because by design the full is taken by the tool in serial instead of in parallel, so let's take a final full by running as postgres =/var/lib/pgsql_backups/dcache-db-backup.sh= ; this full backup is going to be copied also into =t3fs05:/swshare/dcache-postgres-backups/t3dcachedb04= where later the Fabio's migration script on =t3dcachedb02= will search it.
   1. Go into =t3dcachedb02:/root/DCACHE-1.9.12-MIGRATION= and run =source ingest.db.backup.sh= to ingest the latest full backup from =t3fs05=.
   1. convert =t3se02= in =t3se01= , check that x509s and conf files are ok, Puppet will do it but still check.
   1. turn on dCache on =t3dcachedb02= and mount =/pnfs=
   1. turn on dCache on =t3se02= and mount =/pnfs=
   1. upgrade dCache on each =t3fs*=, remember that you have to change the owner of pools files and dirs from =root= to =dcache=, [[http://www.dcache.org/manuals/Book-1.9.12/Book.shtml#in-install-non-root][REF]] . Consider =t3fs*:/root/dcache.conf= and layout =t3fs*:/root/t3fs*.conf=
UpgradePlanningForm
Title	Yearly Shutdown and dCache upgrade to 1.9.12
Summary	shutting down of all services for yearly PSI shutdown. The systems can stay up, but network is lost for some time on Jan 12th. Upgrade of dcache to 1.9.12 on Jan 14.
Target Date	11. 01. 2013