Node Type: WNsSuperMicro

Firewall requirements

local port open to reason


Regular Maintenance work

Nagios

check UIs on t3nagios

User quota on /tmp /scratch ( but not super important )

When a new T3 user is created you have to run on each UI :
# kinit -k -t /root/afs-keytabs/svcusr-t3_puppet.keytab svcusr-t3_puppet@D.PSI.CH && aklog && ls -l /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests && puppet apply --environment DerekDevelopment /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests --modulepath=/afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules --show_diff --color=false

SuperMicro IPMI tools

In-Band and Out-Band tools

https://www.supermicro.com/products/nfo/SMS_IPMI.cfm

Out-Band tools on t3admin

total 187968
lrwxrwxrwx 1 root root       27 Nov 21 14:41 IPMICFG -> IPMICFG_1.25.0_build.160715
drwxrwx--- 5 root root     4096 Nov 21 14:41 IPMICFG_1.25.0_build.160715
-rw-r----- 1 root root  1571697 Nov 21 12:00 IPMICFG_1.25.0_build.160715.zip
drwxr-x--- 4 root root     4096 Nov 18 22:33 IPMI_FIRMWARE
lrwxrwxrwx 1 root root       48 Nov 21 14:53 IPMIView -> IPMIView_2.12.0_build.160804_bundleJRE_Linux_x64
drwxr-x--- 4 root root     4096 Nov 21 15:00 IPMIView_2.12.0_build.160804_bundleJRE_Linux_x64
-rw-r----- 1 root root 91078080 Nov 21 14:43 IPMIView_2.12.0_build.160804_bundleJRE_Linux_x64.tar.gz
drwxr-x--- 2 root root     4096 Nov 18 22:34 IPMIView_2.9.23_jar_20130709
-rw-r----- 1 root root 27155569 Jun 18  2014 IPMIView_2.9.23_jar_20130709.zip
lrwxrwxrwx 1 root root       51 Nov 21 15:06 SMCIPMITool -> SMCIPMITool_2.16.0_build.160816_bundleJRE_Linux_x64
drwxr-x--- 4 root root     4096 Nov 21 14:43 SMCIPMITool_2.16.0_build.160816_bundleJRE_Linux_x64
-rw-r----- 1 root root 72440975 Nov 21 14:36 SMCIPMITool_2.16.0_build.160816_bundleJRE_Linux_x64.tar.gz
lrwxrwxrwx 1 root root        7 Nov 21 14:41 to_be_used_locally_on_the_t3wn_servers -> IPMICFG

In-Band tool on t3wn[41-50]

  • /opt/SUPERMICRO/IPMICFG can be used to totally reset the BMC if that's got stuck during an IPMI FW upgrade process ( seldom occurred though )
  • [root@t3wn41 64bit]# ./IPMICFG-Linux.x86_64 -fde     
    Reset to the factory default completed.
    
  • IPMICFG
    
    IPMICFG is an in-band utility for configuring IPMI devices. It is a command line tool providing standard 
    IPMI and Supermicro proprietary OEM commands. This CLI-based utility can be executed on DOS, 
    Windows, and Linux OS and does not require any installation procedures. 
    This utility is used for BMC/FRU configuration. IPMICFG Key Features:
      - Set up IPMI IP Address
      - Set up IPMI Configuration
      - Configure IPMI User Management
      - Configure IPMI FRU
      - Manage System Event Log (SEL)
      - Manage IPMI by node management (NM) protocol 
    
  • IPMICFG_UserGuide.pdf

Updating the SuperMicro IPMI FW itself ( seldom )

Emergency Measures

The WNs servers run on the top of a mdadm raid10, so they can survive to 1 broken disk ; to repair online the broken disk simply adapt this procedure NodeTypeWNsIntelS2600JF#Degraded_RAID1_or_Failed_RAID0 to these servers. Generally speaking all the mdadm operations apply.

Installation

Because of the mdadm raid10 we should never get to the point where a reinstallation is needed ; anyhow the Puppet recipes are in /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests :

  • SL6_wn.pp
  • SL6.pp
  • tier3-baseclasses.pp
  • and the partition/raid10 layouts are /afs/psi.ch/software/linux/dist/scientific/66/x86_64/custom/tier3/t3wn/ks-partition.cfg

Fabio uses these alias :

alias ROOT='. /afs/cern.ch/sw/lcg/external/gcc/4.8/x86_64-slc6/setup.sh && . /afs/cern.ch/sw/lcg/app/releases/ROOT/5.34.26/x86_64-slc6-gcc48-opt/root/bin/thisroot.sh'
alias cscsela='ssh -AX fmartine@ela.cscs.ch'
alias cscslogin='ssh -AX fmartine@login.lcg.cscs.ch'
alias cscspub='ssh -AX fmartinelli@pub.lcg.cscs.ch'
alias dcache='ssh -2 -l admin -p 22224 t3dcachedb.psi.ch'
alias dcache04='ssh -2 -l admin -p 22224 t3dcachedb04.psi.ch'
alias gempty='git commit --allow-empty-message -m '\'''\'''
alias kscustom54='cd /afs/psi.ch/software/linux/dist/scientific/54/custom'
alias kscustom57='cd /afs/psi.ch/software/linux/dist/scientific/57/custom'
alias kscustom60='cd /afs/psi.ch/software/linux/dist/scientific/60/custom'
alias kscustom64='cd /afs/psi.ch/software/linux/dist/scientific/64/custom'
alias kscustom66='cd /afs/psi.ch/software/linux/dist/scientific/66/x86_64/custom'
alias ksdir='cd /afs/psi.ch/software/linux/kickstart/configs'
alias ksprepostdir='cd /afs/psi.ch/software/linux/dist/scientific/60/kickstart/bin'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=tty'
alias mc='. /usr/libexec/mc/mc-wrapper.sh'
alias pdir='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/'
alias pdirf='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/FabioDevelopment/'
alias pdirmanifests='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/manifests/'
alias pdirredhat='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/Tier3/files/RedHat'
alias pdirsolaris='cd /afs/psi.ch/service/linux/puppet/var/puppet/environments/DerekDevelopment/modules/Tier3/files/Solaris/5.10'
alias vi='vim'
alias which='alias | /usr/bin/which --tty-only --read-alias --show-dot --show-tilde'
alias yumdir5='cd /afs/psi.ch/software/linux/dist/scientific/57/scripts'
alias yumdir6='cd /afs/psi.ch/software/linux/dist/scientific/6/scripts'
alias yumdir7='cd /afs/psi.ch/software/linux/dist/scientificlinux/7x/x86_64/Tier3/all'
alias yumdir7old='cd /afs/psi.ch/software/linux/dist/scientific/70.PLEASE_DO_NOT_USE_AND_DO_NOT_RENAME/scripts'

Services

Basically only SSHd.

netstat -tupl

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 *:7937                      *:*                         LISTEN      7720/nsrexecd       
tcp        0      0 *:7938                      *:*                         LISTEN      7720/nsrexecd       
tcp        0      0 *:5666                      *:*                         LISTEN      7652/nrpe           
tcp        0      0 *:40291                     *:*                         LISTEN      -                   
tcp        0      0 *:secure-ts                 *:*                         LISTEN      7720/nsrexecd       
tcp        0      0 localhost:smux              *:*                         LISTEN      7188/snmpd          
tcp        0      0 *:37480                     *:*                         LISTEN      -                   
tcp        0      0 *:sge_execd                 *:*                         LISTEN      7170/sge_execd      
tcp        0      0 *:sunrpc                    *:*                         LISTEN      2730/rpcbind        
tcp        0      0 *:8146                      *:*                         LISTEN      7720/nsrexecd       
tcp        0      0 *:ssh                       *:*                         LISTEN      7204/sshd           
tcp        0      0 *:46838                     *:*                         LISTEN      2750/rpc.statd      
tcp        0      0 localhost:x11-ssh-offset    *:*                         LISTEN      10728/sshd          
udp        0      0 *:mdns                      *:*                                     6882/avahi-daemon   
udp        0      0 *:sunrpc                    *:*                                     2730/rpcbind        
udp        0      0 *:46194                     *:*                                     6882/avahi-daemon   
udp        0      0 *:ipp                       *:*                                     2597/portreserve    
udp        0      0 t3wn41.psi.ch:ntp           *:*                                     7214/ntpd           
udp        0      0 localhost:ntp               *:*                                     7214/ntpd           
udp        0      0 *:ntp                       *:*                                     7214/ntpd           
udp        0      0 *:7938                      *:*                                     7720/nsrexecd       
udp        0      0 localhost:syslog            *:*                                     2618/syslog-ng      
udp        0      0 *:781                       *:*                                     2730/rpcbind        
udp        0      0 *:snmp                      *:*                                     7188/snmpd          
udp        0      0 *:54693                     *:*                                     2750/rpc.statd      
udp        0      0 localhost:806               *:*                                     2750/rpc.statd      
udp        0      0 *:48426                     *:*                                     -                   
udp        0      0 *:bootpc                    *:*                                     2540/dhclient       
udp        0      0 *:afs3-callback             *:*                                     -  

Backups

Not needed.
NodeTypeForm
Hostnames t3wn[41-50]
Services Sun Grid Engine 6.2u5 execution hosts
Hardware SuperMicro 1u got from CSCS
Install Profile t3wn
Guarantee/maintenance until out of warranty
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf IPMICFG_UserGuide.pdf r1 manage 677.3 K 2016-11-21 - 11:18 FabioMartinelli SuperMicro In-Band management on Linux
Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r4 - 2016-11-21 - FabioMartinelli
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback