Firewall requirements
Regular Maintenance work
Nagios
check UIs on t3nagios
User quota on /tmp /scratch
( but not super important )
When a new T3 user is created you have to run on each UI :
# kinit -k -t /root/afs-keytabs/svcusr-t3_puppet.keytab svcusr-t3_puppet@D.PSI.CH && aklog && ls -l /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests && puppet apply --environment DerekDevelopment /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests --modulepath=/afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules --show_diff --color=false
In-Band and Out-Band tools
https://www.supermicro.com/products/nfo/SMS_IPMI.cfm
Out-Band tools on t3admin
total 187968
lrwxrwxrwx 1 root root 27 Nov 21 14:41 IPMICFG -> IPMICFG_1.25.0_build.160715
drwxrwx--- 5 root root 4096 Nov 21 14:41 IPMICFG_1.25.0_build.160715
-rw-r----- 1 root root 1571697 Nov 21 12:00 IPMICFG_1.25.0_build.160715.zip
drwxr-x--- 4 root root 4096 Nov 18 22:33 IPMI_FIRMWARE
lrwxrwxrwx 1 root root 48 Nov 21 14:53 IPMIView -> IPMIView_2.12.0_build.160804_bundleJRE_Linux_x64
drwxr-x--- 4 root root 4096 Nov 21 15:00 IPMIView_2.12.0_build.160804_bundleJRE_Linux_x64
-rw-r----- 1 root root 91078080 Nov 21 14:43 IPMIView_2.12.0_build.160804_bundleJRE_Linux_x64.tar.gz
drwxr-x--- 2 root root 4096 Nov 18 22:34 IPMIView_2.9.23_jar_20130709
-rw-r----- 1 root root 27155569 Jun 18 2014 IPMIView_2.9.23_jar_20130709.zip
lrwxrwxrwx 1 root root 51 Nov 21 15:06 SMCIPMITool -> SMCIPMITool_2.16.0_build.160816_bundleJRE_Linux_x64
drwxr-x--- 4 root root 4096 Nov 21 14:43 SMCIPMITool_2.16.0_build.160816_bundleJRE_Linux_x64
-rw-r----- 1 root root 72440975 Nov 21 14:36 SMCIPMITool_2.16.0_build.160816_bundleJRE_Linux_x64.tar.gz
lrwxrwxrwx 1 root root 7 Nov 21 14:41 to_be_used_locally_on_the_t3wn_servers -> IPMICFG
In-Band tool on t3wn[41-50]
-
/opt/SUPERMICRO/IPMICFG
can be used to totally reset the BMC if that's got stuck during an IPMI FW upgrade process ( seldom occurred though )
-
[root@t3wn41 64bit]# ./IPMICFG-Linux.x86_64 -fde
Reset to the factory default completed.
-
IPMICFG
IPMICFG is an in-band utility for configuring IPMI devices. It is a command line tool providing standard
IPMI and SupermicroŽ proprietary OEM commands. This CLI-based utility can be executed on DOS,
Windows, and Linux OS and does not require any installation procedures.
This utility is used for BMC/FRU configuration. IPMICFG Key Features:
- Set up IPMI IP Address
- Set up IPMI Configuration
- Configure IPMI User Management
- Configure IPMI FRU
- Manage System Event Log (SEL)
- Manage IPMI by node management (NM) protocol
- IPMICFG_UserGuide.pdf
Updating the SuperMicro IPMI FW itself ( seldom )
Emergency Measures
The WNs servers run on the top of a
mdadm raid10
, so they can survive to 1 broken disk ; to repair
online the broken disk simply adapt this procedure
NodeTypeWNsIntelS2600JF#Degraded_RAID1_or_Failed_RAID0 to these servers. Generally speaking all the
mdadm
operations apply.
Installation
Because of the
mdadm raid10
we should never get to the point where a reinstallation is needed ; anyhow the Puppet recipes are in
/afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests
:
-
SL6_wn.pp
-
SL6.pp
-
tier3-baseclasses.pp
- and the partition/raid10 layouts are
/afs/psi.ch/software/linux/dist/scientific/66/x86_64/custom/tier3/t3wn/ks-partition.cfg
Fabio uses these alias :
alias ROOT='. /afs/cern.ch/sw/lcg/external/gcc/4.8/x86_64-slc6/setup.sh && . /afs/cern.ch/sw/lcg/app/releases/ROOT/5.34.26/x86_64-slc6-gcc48-opt/root/bin/thisroot.sh'
alias cscsela='ssh -AX fmartine@ela.cscs.ch'
alias cscslogin='ssh -AX fmartine@login.lcg.cscs.ch'
alias cscspub='ssh -AX fmartinelli@pub.lcg.cscs.ch'
alias dcache='ssh -2 -l admin -p 22224 t3dcachedb.psi.ch'
alias dcache04='ssh -2 -l admin -p 22224 t3dcachedb04.psi.ch'
alias gempty='git commit --allow-empty-message -m '\'''\'''
alias kscustom54='cd /afs/psi.ch/software/linux/dist/scientific/54/custom'
alias kscustom57='cd /afs/psi.ch/software/linux/dist/scientific/57/custom'
alias kscustom60='cd /afs/psi.ch/software/linux/dist/scientific/60/custom'
alias kscustom64='cd /afs/psi.ch/software/linux/dist/scientific/64/custom'
alias kscustom66='cd /afs/psi.ch/software/linux/dist/scientific/66/x86_64/custom'
alias ksdir='cd /afs/psi.ch/software/linux/kickstart/configs'
alias ksprepostdir='cd /afs/psi.ch/software/linux/dist/scientific/60/kickstart/bin'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=tty'
alias mc='. /usr/libexec/mc/mc-wrapper.sh'
alias pdir='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/'
alias pdirf='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/FabioDevelopment/'
alias pdirmanifests='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/'
alias pdirredhat='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/Tier3/files/RedHat'
alias pdirsolaris='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/Tier3/files/Solaris/5.10'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'
alias yumdir5='cd /afs/psi.ch/software/linux/dist/scientific/57/scripts'
alias yumdir6='cd /afs/psi.ch/software/linux/dist/scientific/6/scripts'
alias yumdir7='cd /afs/psi.ch/software/linux/dist/scientificlinux/7x/x86_64/Tier3/all'
alias yumdir7old='cd /afs/psi.ch/software/linux/dist/scientific/70.PLEASE_DO_NOT_USE_AND_DO_NOT_RENAME/scripts'
Services
Basically only SSHd.
netstat -tupl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:7937 *:* LISTEN 7720/nsrexecd
tcp 0 0 *:7938 *:* LISTEN 7720/nsrexecd
tcp 0 0 *:5666 *:* LISTEN 7652/nrpe
tcp 0 0 *:40291 *:* LISTEN -
tcp 0 0 *:secure-ts *:* LISTEN 7720/nsrexecd
tcp 0 0 localhost:smux *:* LISTEN 7188/snmpd
tcp 0 0 *:37480 *:* LISTEN -
tcp 0 0 *:sge_execd *:* LISTEN 7170/sge_execd
tcp 0 0 *:sunrpc *:* LISTEN 2730/rpcbind
tcp 0 0 *:8146 *:* LISTEN 7720/nsrexecd
tcp 0 0 *:ssh *:* LISTEN 7204/sshd
tcp 0 0 *:46838 *:* LISTEN 2750/rpc.statd
tcp 0 0 localhost:x11-ssh-offset *:* LISTEN 10728/sshd
udp 0 0 *:mdns *:* 6882/avahi-daemon
udp 0 0 *:sunrpc *:* 2730/rpcbind
udp 0 0 *:46194 *:* 6882/avahi-daemon
udp 0 0 *:ipp *:* 2597/portreserve
udp 0 0 t3wn41.psi.ch:ntp *:* 7214/ntpd
udp 0 0 localhost:ntp *:* 7214/ntpd
udp 0 0 *:ntp *:* 7214/ntpd
udp 0 0 *:7938 *:* 7720/nsrexecd
udp 0 0 localhost:syslog *:* 2618/syslog-ng
udp 0 0 *:781 *:* 2730/rpcbind
udp 0 0 *:snmp *:* 7188/snmpd
udp 0 0 *:54693 *:* 2750/rpc.statd
udp 0 0 localhost:806 *:* 2750/rpc.statd
udp 0 0 *:48426 *:* -
udp 0 0 *:bootpc *:* 2540/dhclient
udp 0 0 *:afs3-callback *:* -
Backups
Not needed.