Tags:
view all tags
<!-- keep this as a security measure: #uncomment if the subject should only be modifiable by the listed groups * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup,Main.CMSAdminReaderGroup --> %TOC% ---+ HP ProLiant DL380 G7 iLO3 ---++ Intro The 2 =HP DL380 G7= servers =t3fs[13,14]= can be managed by the specific HP Service Processor called [[www.hp.com/servers/lights-out][iLO3]] that basically is a dedicated Ethernet port on the rear of the server providing the following management mechanisms: * by =IPMI= * by =SSH= + SL6 cooperation to get the *Serial* Virtual Console ( it's like to attach a serial cable to the server ) * by =HTTPS= + Java/.NET plugins in your browser to get the *Remote* Virtual Console ( it's like to be in front of the VGA screen of the server, so you can run also graphic commands ) For both server that port is connected to our private 100Mbit/s network 192.168.2.21/24 that's reacheable only through =t3admin01=, so first you'll need to login into =t3admin01= and then interact with =iLO3=. You can also configure =iLO3= by rebooting the server and entering into the iLO3 BIOS but that's usually a mechanism you won't use because the servers are in production. Once properly configured and understood you can perform by =iLO3= many common Sys Admin operations like: * Redirect the console * Read HW sensors * Apply power on/off/reset cycles * Mount a Linux recovery CD/DVD .iso to recover a corrupted installation ---++ Users When I've mounted in the rack the 2 servers I was asking to myself how to login into =iLO3=, eventually on the front of the each server there is a =(i)= flag that you can pull out and read the HP factory id/pwd, but because I didn't notice it I've also created by =iLO3 BIOS= an user =root= with the Admin privileges; actually I still prefer this =root= user I made because it's a common user inside our private network and thus we can forget about the specific =iLO3 user=. ---++ IPMI ---+++ IPMI daemon First check out if the IPMI daemon is listening on UDP: <pre> [root@t3admin01 sbin]# nmap -sU rmfs13 -p 623 Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2013-10-15 16:53 CEST Interesting ports on rmfs13 (192.168.2.55): PORT STATE SERVICE 623/udp %BLUE%open%ENDCOLOR%|filtered unknown MAC Address: 44:1E:A1:3A:46:B4 (Unknown) Nmap finished: 1 IP address (1 host up) scanned in 0.310 seconds </pre> ---+++ ipmitool Pay attention to that *-I lanplus* option ! <pre> [root@t3admin01 ~]# ipmitool %BLUE%-I lanplus%ENDCOLOR% -H rmfs13 -U root -f /root/private/ipmi-pw sdr UID Light | 0 unspecified | ok Sys. Health LED | 0 unspecified | ok Power Supply 1 | 65 Watts | nc Power Supply 2 | 85 Watts | nc Power Supplies | 0 unspecified | nc Fan 1 | 29.40 unspecifi | nc Fan 2 | 29.40 unspecifi | nc Fan 3 | 46.26 unspecifi | nc Fan 4 | 54.49 unspecifi | nc Fan 5 | 54.49 unspecifi | nc Fan 6 | 13.72 unspecifi | nc Fans | 0 unspecified | nc Temp 1 | 19 degrees C | ok Temp 2 | 40 degrees C | ok Temp 3 | 40 degrees C | ok Temp 4 | 31 degrees C | ok Temp 5 | 33 degrees C | ok Temp 6 | 29 degrees C | ok Temp 7 | 33 degrees C | ok Temp 8 | 35 degrees C | ok Temp 9 | 30 degrees C | ok Temp 10 | 40 degrees C | ok Temp 11 | 32 degrees C | ok Temp 12 | 40 degrees C | ok Temp 13 | 27 degrees C | ok Temp 14 | 31 degrees C | ok Temp 15 | 31 degrees C | ok Temp 16 | disabled | ns Temp 17 | disabled | ns Temp 18 | disabled | ns Temp 19 | 25 degrees C | ok Temp 20 | 26 degrees C | ok Temp 21 | 28 degrees C | ok Temp 22 | 28 degrees C | ok Temp 23 | 33 degrees C | ok Temp 24 | 31 degrees C | ok Temp 25 | 27 degrees C | ok Temp 26 | 28 degrees C | ok Temp 27 | 32 degrees C | ok Temp 28 | disabled | ns Temp 29 | 35 degrees C | ok Temp 30 | 59 degrees C | ok Memory | 0 error | ok Power Meter | 146 Watts | cr Cntlr 1 Bay 1 | 0 unspecified | nc Cntlr 1 Bay 2 | 0 unspecified | nc Cntlr 1 Bay 3 | 0 unspecified | ok Cntlr 1 Bay 4 | 0 unspecified | ok Cntlr 2 Bay 5 | 0 unspecified | ok Cntlr 2 Bay 6 | 0 unspecified | ok Cntlr 2 Bay 7 | 0 unspecified | ok Cntlr 2 Bay 8 | 0 unspecified | ok </pre> ---+++ freeipmi sensors <pre> [root@t3admin01 sbin]# /usr/sbin/ipmi-sensors -h rmfs13 %BLUE%-D lan_2_0%ENDCOLOR% --config-file /opt/nagios/check_ipmi_sensor.user.pwd.privilege -t Temperature ID | Name | Type | Reading | Units | Event 14 | Temp 1 | Temperature | 19.00 | C | 'OK' 15 | Temp 2 | Temperature | 40.00 | C | 'OK' 16 | Temp 3 | Temperature | 40.00 | C | 'OK' 17 | Temp 4 | Temperature | 28.00 | C | 'OK' 18 | Temp 5 | Temperature | 30.00 | C | 'OK' 19 | Temp 6 | Temperature | 29.00 | C | 'OK' 20 | Temp 7 | Temperature | 32.00 | C | 'OK' 21 | Temp 8 | Temperature | 35.00 | C | 'OK' 22 | Temp 9 | Temperature | 29.00 | C | 'OK' 23 | Temp 10 | Temperature | 40.00 | C | 'OK' 24 | Temp 11 | Temperature | 31.00 | C | 'OK' 25 | Temp 12 | Temperature | 39.00 | C | 'OK' 26 | Temp 13 | Temperature | 26.00 | C | 'OK' 27 | Temp 14 | Temperature | 30.00 | C | 'OK' 28 | Temp 15 | Temperature | 30.00 | C | 'OK' 29 | Temp 16 | Temperature | N/A | C | N/A 30 | Temp 17 | Temperature | N/A | C | N/A 31 | Temp 18 | Temperature | N/A | C | N/A 32 | Temp 19 | Temperature | 25.00 | C | 'OK' 33 | Temp 20 | Temperature | 26.00 | C | 'OK' 34 | Temp 21 | Temperature | 28.00 | C | 'OK' 35 | Temp 22 | Temperature | 27.00 | C | 'OK' 36 | Temp 23 | Temperature | 32.00 | C | 'OK' 37 | Temp 24 | Temperature | 31.00 | C | 'OK' 38 | Temp 25 | Temperature | 27.00 | C | 'OK' 39 | Temp 26 | Temperature | 28.00 | C | 'OK' 40 | Temp 27 | Temperature | 31.00 | C | 'OK' 41 | Temp 28 | Temperature | N/A | C | N/A 42 | Temp 29 | Temperature | 35.00 | C | 'OK' 43 | Temp 30 | Temperature | 58.00 | C | 'OK' </pre> ---+++ Nagios check based on freeipmi sensors <pre> [root@t3admin01 sbin]# /opt/nagios/check_ipmi_sensor %BLUE%-O '-D lan_2_0 -t Temperature'%ENDCOLOR% -f /opt/nagios/check_ipmi_sensor.user.pwd.privilege -H rmfs13 %BLUE%IPMI Status: OK%ENDCOLOR% | 'Temp 1'=19.00 'Temp 2'=40.00 'Temp 3'=40.00 'Temp 4'=28.00 'Temp 5'=30.00 'Temp 6'=29.00 'Temp 7'=32.00 'Temp 8'=35.00 'Temp 9'=29.00 'Temp 10'=40.00 'Temp 11'=31.00 'Temp 12'=39.00 'Temp 13'=26.00 'Temp 14'=30.00 'Temp 15'=30.00 'Temp 19'=25.00 'Temp 20'=26.00 'Temp 21'=28.00 'Temp 22'=27.00 'Temp 23'=32.00 'Temp 24'=31.00 'Temp 25'=27.00 'Temp 26'=28.00 'Temp 27'=31.00 'Temp 29'=35.00 'Temp 30'=58.00 </pre> ---++ SSH Be aware that you can even install your daily SSH PUB key into ILO, this will simplify your logins: <pre>[root@t3admin01 ~]# ssh rmfs13 User:root logged-in to ILOCZ31513SBR.(192.168.2.55) iLO 3 Advanced 1.26 at Aug 26 2011 Server Name: t3fs13 Server Power: On hpiLO-> show status=0 status_tag=COMMAND COMPLETED Tue Feb 14 11:19:54 2012 / Targets system1 map1 Properties Verbs cd version exit show </pre> Basically the =/system1= directory is an abstraction to read/manage the server parameters like for instance request a server reboot, read sensors, and so on while the directory =/map1= models the specific =iLO3= parameters, for instance you might reset the =iLO3= module itself: the word =Verbs= lists the actions you can perform inside a directory. The simplest action is to jump into an other directory and list the available commands for that directory: <pre>hpiLO-> cd /map1 /map1 hpiLO-> show /map1 Targets firmware1 accounts1 log1 enetport1 dhcpendpt1 dnsendpt1 gateway1 dnsserver1 dnsserver2 dnsserver3 dhcpserver1 settings1 config1 snmp1 oemhp_dircfg1 oemhp_vm1 vlan1 oemhp_ssocfg1 Properties name=iLO 3 Advanced license=35SCT5KKSJ5GXVDZM75KPW8ZH Verbs cd version exit show reset set oemhp_ping hpiLO-> show config1 /map1/config1 Targets Properties oemhp_mapenable=yes oemhp_timeout=30 oemhp_rbsuenable=yes oemhp_rbsulogin=no oemhp_rbsushowip=yes oemhp_httpport=80 oemhp_sslport=443 oemhp_rcport=17990 oemhp_vmport=17988 oemhp_sshport=22 oemhp_sshstatus=yes oemhp_serialclistatus=yes oemhp_serialcliauth=no oemhp_serialclispeed=115200 oemhp_minpwdlen=8 oemhp_enforce_aes=no oemhp_authfailurelogging=3 oemhp_computer_lock=disabled Verbs cd version exit show set oemhp_loadSSHkey hpiLO-> </pre> ---+++ Serial Virtual Console ---++++ SL6 cooperation One of the most useful commands you get from =iLO3= is the serial redirection of the Linux console like in this example: <pre>hpiLO-> vsp Virtual Serial Port Active: COM2 Starting virtual serial port. Press 'ESC (' to return to the CLI Session. Scientific Linux release 6.0 (Carbon) Kernel 2.6.32-220.4.1.el6.x86_64 on an x86_64 t3fs13.psi.ch login: </pre> Like adviced, by pressing the keys combinations =ESC + SHIFT + (= you will exit from the Serial Virtual Console; please notice that =COM2= maps in Linux to =ttyS1= and it's explicitely reported in these 2 files: <pre>[root@t3fs14 sdb]# grep S1 /etc/securetty ttyS1 [root@t3fs14 sdb]# grep S1 /boot/grub/menu.lst kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 ro root=UUID=a247aed2-7b16-4306-9485-2adc3f62a6da rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=ttyS1,115200 crashkernel=auto elevator=deadline </pre> Indeed since SL6 it's no more needed to write lines like this: <pre># grep S1 /etc/inittab T0:2345:respawn:/sbin/agetty -L -i 115200 ttyS1 vt102 </pre> to redirect the console; SL6 will take into account the grub parameter =console= and it will activate the permanent redirection. By using the Serial Virtual Console you'll be able to see even the HW initialization phase when you'll reboot these servers and later the =GRUB= dialog. ---++++ SL6 installation The Serial Virtual Console is also useful during an SL6 installation but again you need the proper cooperation from SL6; to achieve that I've modified our Kickstart installation parameters like showed: <pre>[root@t3admin01 pxelinux.cfg]# cat default default localboot prompt 1 timeout 300 display boot.msg F1 boot.msg ################################ # Local Boot (default) ################################ label localboot LOCALBOOT 0 ###################################### # Scientific Linux 6 2012-02 by martinelli ###################################### label sl60 kernel scientific/6x/x86_64/vmlinuz append text initrd=scientific/6x/x86_64/initrd.img ks=http://linux.web.psi.ch/kickstart/configs/sl60-64-tier3-ks.cfg noipv6 console=ttyS1,115200 ksdevice=eth2 cmdline 0 ... </pre> Please pay attention to those =append= parameters also described inside the [[http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Installation_Guide/index.html#ch-parmfiles-Kickstart_parameters][official installation guide]]: * =ksdevice= : is used to explicitly cite the Ethernet port to be used during the Kickstart installation, so =Anaconda= can skip the question about that ( without that option =Anaconda= was stopping and asking ) * =cmdline= : When =cmdline= is specified, output on line-mode terminals (such as 3270 under z/VM or operating system messages for LPAR) becomes readable, as the installer disables escape terminal sequences that are only applicable to =UNIX-like= consoles. This requires installation with a kickstart file that answers all questions, since the installer does not support interactive user input in =cmdline= mode. ---+++ Firmware version <pre> </map1>hpiLO-> show /map1/firmware1 status=0 status_tag=COMMAND COMPLETED Tue Mar 20 15:32:26 2012 /map1/firmware1 Targets Properties version=1.26 date=Aug 26 2011 Verbs cd version exit show load set </pre> ---+++ Firmware update made on 27th Dec '13 <pre> [root@t3fs13 CP016203]# ./flash_ilo3 FLASH_iLO3 v1.08 for Linux (May 8 2012) (C) Copyright 2002-2012 Hewlett-Packard Development Company, L.P. Firmware image: ilo3_150.bin Current iLO 3 firmware version 1.26; Serial number ILOCZ31513SBR Component XML file: CP016203.xml CP016203.xml reports firmware version 1.50 This operation will update the firmware on the iLO 3 in this server with version 1.50. Continue (y/N)?y Current firmware is 1.26 (Aug 26 2011 ) Firmware image is 0x801664(8394340) bytes Committing to flash part... ******** DO NOT INTERRUPT! ******** Flashing completed. Attempting to reset device. Succeeded. ***** iLO 3 reboot in progress (may take up to 60 seconds.) ***** Please ignore console messages, if any. iLO 3 reboot completed. </pre> ---+++ Events LOGs - 1 record example showed <pre> </map1>hpiLO-> show /map1/log1/record452 status=0 status_tag=COMMAND COMPLETED Tue Mar 20 15:36:23 2012 /map1/log1/record452 Targets Properties number=452 severity=Informational date=03/20/2012 time=15:22 description=SSH login: root - 192.168.2.21(DNS name not found). Verbs cd version exit show set </pre> ---++ HTTPS The simplest mechansim to interact with iLO3 is by starting firefox, like: <pre>firefox rmfs14 &</pre> ---+++ Remote Virtual Console You can access the remote video of the server by firefox + Java but obviously the Java plugin wasn't configured inside the firefox installed in =t3admin01= :-), for install it you just need to make a symbolic link like here showed: <pre>[root@t3admin01 plugins]# pwd /usr/lib64/firefox-3.6/plugins [root@t3admin01 plugins]# ll total 0 lrwxrwxrwx 1 root root 46 Feb 1 15:02 libjava.so -> /usr/java/jdk1.6.0_29/jre/lib/amd64/libjava.so </pre> It's worth to mention that *by using the Java Remote Virtual Console you can switch among the several screens that Linux provides*, basically you'll find the equivalent of =ALT+F[1-8]=, those are very useful during a Linux installation. ---+ [[NodeTypeFileServerHP][dCache role of these servers]]
Edit
|
Attach
|
Watch
|
P
rint version
|
H
istory
:
r9
<
r8
<
r7
<
r6
<
r5
|
B
acklinks
|
V
iew topic
|
Raw edit
|
More topic actions...
Topic revision: r7 - 2013-10-16
-
FabioMartinelli
CmsTier3
Log In
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
User Pages
Main Page
Policies
Monitoring Storage Space
Monitoring Slurm Usage
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
Edit
Attach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback