HP ProLiant DL380 G7 iLO3
Intro
The 2
HP DL380 G7
servers
t3fs[13,14]
can be managed by the specific HP Service Processor called
iLO3 that basically is a dedicated Ethernet port on the rear of the server providing the following management mechanisms:
- by
IPMI
- by
SSH
+ SL6 cooperation to get the Serial Virtual Console ( it's like to attach a serial cable to the server )
- by
HTTPS
+ Java/.NET plugins in your browser to get the Remote Virtual Console ( it's like to be in front of the VGA screen of the server, so you can run also graphic commands )
For both server that port is connected to our private 100Mbit/s network 192.168.2.21/24 that's reacheable only through
t3admin01
, so first you'll need to login into
t3admin01
and then interact with
iLO3
.
You can also configure
iLO3
by rebooting the server and entering into the iLO3 BIOS but that's usually a mechanism you won't use because the servers are in production.
Once properly configured and understood you can perform by
iLO3
many common Sys Admin operations like:
- Redirect the console
- Read HW sensors
- Apply power on/off/reset cycles
- Mount a Linux recovery CD/DVD .iso to recover a corrupted installation
Users
When I've mounted in the rack the 2 servers I was asking to myself how to login into
iLO3
, eventually on the front of the each server there is a
(i)
flag that you can pull out and read the HP factory id/pwd, but because I didn't notice it I've also created by
iLO3 BIOS
an user
root
with the Admin privileges; actually I still prefer this
root
user I made because it's a common user inside our private network and thus we can forget about the specific
iLO3 user
.
IPMI
IPMI daemon
First check out if the IPMI daemon is listening on UDP:
[root@t3admin01 sbin]# nmap -sU rmfs13 -p 623
Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2013-10-15 16:53 CEST
Interesting ports on rmfs13 (192.168.2.55):
PORT STATE SERVICE
623/udp open|filtered unknown
MAC Address: 44:1E:A1:3A:46:B4 (Unknown)
Nmap finished: 1 IP address (1 host up) scanned in 0.310 seconds
ipmitool
Pay attention to that
-I lanplus option !
[root@t3admin01 ~]# ipmitool -I lanplus -H rmfs13 -U root -f /root/private/ipmi-pw sdr
UID Light | 0 unspecified | ok
Sys. Health LED | 0 unspecified | ok
Power Supply 1 | 65 Watts | nc
Power Supply 2 | 85 Watts | nc
Power Supplies | 0 unspecified | nc
Fan 1 | 29.40 unspecifi | nc
Fan 2 | 29.40 unspecifi | nc
Fan 3 | 46.26 unspecifi | nc
Fan 4 | 54.49 unspecifi | nc
Fan 5 | 54.49 unspecifi | nc
Fan 6 | 13.72 unspecifi | nc
Fans | 0 unspecified | nc
Temp 1 | 19 degrees C | ok
Temp 2 | 40 degrees C | ok
Temp 3 | 40 degrees C | ok
Temp 4 | 31 degrees C | ok
Temp 5 | 33 degrees C | ok
Temp 6 | 29 degrees C | ok
Temp 7 | 33 degrees C | ok
Temp 8 | 35 degrees C | ok
Temp 9 | 30 degrees C | ok
Temp 10 | 40 degrees C | ok
Temp 11 | 32 degrees C | ok
Temp 12 | 40 degrees C | ok
Temp 13 | 27 degrees C | ok
Temp 14 | 31 degrees C | ok
Temp 15 | 31 degrees C | ok
Temp 16 | disabled | ns
Temp 17 | disabled | ns
Temp 18 | disabled | ns
Temp 19 | 25 degrees C | ok
Temp 20 | 26 degrees C | ok
Temp 21 | 28 degrees C | ok
Temp 22 | 28 degrees C | ok
Temp 23 | 33 degrees C | ok
Temp 24 | 31 degrees C | ok
Temp 25 | 27 degrees C | ok
Temp 26 | 28 degrees C | ok
Temp 27 | 32 degrees C | ok
Temp 28 | disabled | ns
Temp 29 | 35 degrees C | ok
Temp 30 | 59 degrees C | ok
Memory | 0 error | ok
Power Meter | 146 Watts | cr
Cntlr 1 Bay 1 | 0 unspecified | nc
Cntlr 1 Bay 2 | 0 unspecified | nc
Cntlr 1 Bay 3 | 0 unspecified | ok
Cntlr 1 Bay 4 | 0 unspecified | ok
Cntlr 2 Bay 5 | 0 unspecified | ok
Cntlr 2 Bay 6 | 0 unspecified | ok
Cntlr 2 Bay 7 | 0 unspecified | ok
Cntlr 2 Bay 8 | 0 unspecified | ok
freeipmi sensors
[root@t3admin01 sbin]# /usr/sbin/ipmi-sensors -h rmfs13 -D lan_2_0 --config-file /opt/nagios/check_ipmi_sensor.user.pwd.privilege -t Temperature
ID | Name | Type | Reading | Units | Event
14 | Temp 1 | Temperature | 19.00 | C | 'OK'
15 | Temp 2 | Temperature | 40.00 | C | 'OK'
16 | Temp 3 | Temperature | 40.00 | C | 'OK'
17 | Temp 4 | Temperature | 28.00 | C | 'OK'
18 | Temp 5 | Temperature | 30.00 | C | 'OK'
19 | Temp 6 | Temperature | 29.00 | C | 'OK'
20 | Temp 7 | Temperature | 32.00 | C | 'OK'
21 | Temp 8 | Temperature | 35.00 | C | 'OK'
22 | Temp 9 | Temperature | 29.00 | C | 'OK'
23 | Temp 10 | Temperature | 40.00 | C | 'OK'
24 | Temp 11 | Temperature | 31.00 | C | 'OK'
25 | Temp 12 | Temperature | 39.00 | C | 'OK'
26 | Temp 13 | Temperature | 26.00 | C | 'OK'
27 | Temp 14 | Temperature | 30.00 | C | 'OK'
28 | Temp 15 | Temperature | 30.00 | C | 'OK'
29 | Temp 16 | Temperature | N/A | C | N/A
30 | Temp 17 | Temperature | N/A | C | N/A
31 | Temp 18 | Temperature | N/A | C | N/A
32 | Temp 19 | Temperature | 25.00 | C | 'OK'
33 | Temp 20 | Temperature | 26.00 | C | 'OK'
34 | Temp 21 | Temperature | 28.00 | C | 'OK'
35 | Temp 22 | Temperature | 27.00 | C | 'OK'
36 | Temp 23 | Temperature | 32.00 | C | 'OK'
37 | Temp 24 | Temperature | 31.00 | C | 'OK'
38 | Temp 25 | Temperature | 27.00 | C | 'OK'
39 | Temp 26 | Temperature | 28.00 | C | 'OK'
40 | Temp 27 | Temperature | 31.00 | C | 'OK'
41 | Temp 28 | Temperature | N/A | C | N/A
42 | Temp 29 | Temperature | 35.00 | C | 'OK'
43 | Temp 30 | Temperature | 58.00 | C | 'OK'
Nagios check based on freeipmi sensors
[root@t3admin01 sbin]# /opt/nagios/check_ipmi_sensor -O '-D lan_2_0 -t Temperature' -f /opt/nagios/check_ipmi_sensor.user.pwd.privilege -H rmfs13
IPMI Status: OK | 'Temp 1'=19.00 'Temp 2'=40.00 'Temp 3'=40.00 'Temp 4'=28.00 'Temp 5'=30.00 'Temp 6'=29.00 'Temp 7'=32.00 'Temp 8'=35.00 'Temp 9'=29.00 'Temp 10'=40.00 'Temp 11'=31.00 'Temp 12'=39.00 'Temp 13'=26.00 'Temp 14'=30.00 'Temp 15'=30.00 'Temp 19'=25.00 'Temp 20'=26.00 'Temp 21'=28.00 'Temp 22'=27.00 'Temp 23'=32.00 'Temp 24'=31.00 'Temp 25'=27.00 'Temp 26'=28.00 'Temp 27'=31.00 'Temp 29'=35.00 'Temp 30'=58.00
SSH
Actually I got a simplest CLI experience by
SSH
, you can login in this way:
[root@t3admin01 ~]# ssh rmfs13
User:root logged-in to ILOCZ31513SBR.(192.168.2.55)
iLO 3 Advanced 1.26 at Aug 26 2011
Server Name: t3fs13
Server Power: On
hpiLO-> show
status=0
status_tag=COMMAND COMPLETED
Tue Feb 14 11:19:54 2012
/
Targets
system1
map1
Properties
Verbs
cd version exit show
Basically the
/system1
directory is an abstraction to read/manage the server parameters like for instance request a server reboot, read sensors, and so on while the directory
/map1
models the specific
iLO3
parameters, for instance you might reset the
iLO3
module itself: the word
Verbs
lists the actions you can perform inside a directory. The simplest action is to jump into an other directory and list the available commands for that directory:
hpiLO-> cd /map1
/map1
hpiLO-> show
/map1
Targets
firmware1
accounts1
log1
enetport1
dhcpendpt1
dnsendpt1
gateway1
dnsserver1
dnsserver2
dnsserver3
dhcpserver1
settings1
config1
snmp1
oemhp_dircfg1
oemhp_vm1
vlan1
oemhp_ssocfg1
Properties
name=iLO 3 Advanced
license=35SCT5KKSJ5GXVDZM75KPW8ZH
Verbs
cd version exit show reset set oemhp_ping
hpiLO-> show config1
/map1/config1
Targets
Properties
oemhp_mapenable=yes
oemhp_timeout=30
oemhp_rbsuenable=yes
oemhp_rbsulogin=no
oemhp_rbsushowip=yes
oemhp_httpport=80
oemhp_sslport=443
oemhp_rcport=17990
oemhp_vmport=17988
oemhp_sshport=22
oemhp_sshstatus=yes
oemhp_serialclistatus=yes
oemhp_serialcliauth=no
oemhp_serialclispeed=115200
oemhp_minpwdlen=8
oemhp_enforce_aes=no
oemhp_authfailurelogging=3
oemhp_computer_lock=disabled
Verbs
cd version exit show set oemhp_loadSSHkey
hpiLO->
Serial Virtual Console
SL6 cooperation
One of the most useful commands you get from
iLO3
is the serial redirection of the Linux console like in this example:
hpiLO-> vsp
Virtual Serial Port Active: COM2
Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.
Scientific Linux release 6.0 (Carbon)
Kernel 2.6.32-220.4.1.el6.x86_64 on an x86_64
t3fs13.psi.ch login:
Like adviced, by pressing the keys combinations
ESC + SHIFT + (
you will exit from the Serial Virtual Console; please notice that
COM2
maps in Linux to
ttyS1
and it's explicitely reported in these 2 files:
[root@t3fs14 sdb]# grep S1 /etc/securetty
ttyS1
[root@t3fs14 sdb]# grep S1 /boot/grub/menu.lst
kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 ro root=UUID=a247aed2-7b16-4306-9485-2adc3f62a6da rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=ttyS1,115200 crashkernel=auto elevator=deadline
Indeed since SL6 it's no more needed to write lines like this:
# grep S1 /etc/inittab
T0:2345:respawn:/sbin/agetty -L -i 115200 ttyS1 vt102
to redirect the console; SL6 will take into account the grub parameter
console
and it will activate the permanent redirection.
By using the Serial Virtual Console you'll be able to see even the HW initialization phase when you'll reboot these servers and later the
GRUB
dialog.
SL6 installation
The Serial Virtual Console is also useful during an SL6 installation but again you need the proper cooperation from SL6; to achieve that I've modified our Kickstart installation parameters like showed:
[root@t3admin01 pxelinux.cfg]# cat default
default localboot
prompt 1
timeout 300
display boot.msg
F1 boot.msg
################################
# Local Boot (default)
################################
label localboot
LOCALBOOT 0
######################################
# Scientific Linux 6 2012-02 by martinelli
######################################
label sl60
kernel scientific/6x/x86_64/vmlinuz
append text initrd=scientific/6x/x86_64/initrd.img ks=http://linux.web.psi.ch/kickstart/configs/sl60-64-tier3-ks.cfg noipv6 console=ttyS1,115200 ksdevice=eth2 cmdline
0
...
Please pay attention to those
append
parameters also described inside the
official installation guide:
-
ksdevice
: is used to explicitly cite the Ethernet port to be used during the Kickstart installation, so Anaconda
can skip the question about that ( without that option Anaconda
was stopping and asking )
-
cmdline
: When cmdline
is specified, output on line-mode terminals (such as 3270 under z/VM or operating system messages for LPAR) becomes readable, as the installer disables escape terminal sequences that are only applicable to UNIX-like
consoles. This requires installation with a kickstart file that answers all questions, since the installer does not support interactive user input in cmdline
mode.
Firmware version
hpiLO-> show /map1/firmware1
status=0
status_tag=COMMAND COMPLETED
Tue Mar 20 15:32:26 2012
/map1/firmware1
Targets
Properties
version=1.26
date=Aug 26 2011
Verbs
cd version exit show load set
Firmware update
made on 27th Dec '13
[root@t3fs13 CP016203]# ./flash_ilo3
FLASH_iLO3 v1.08 for Linux (May 8 2012)
(C) Copyright 2002-2012 Hewlett-Packard Development Company, L.P.
Firmware image: ilo3_150.bin
Current iLO 3 firmware version 1.26; Serial number ILOCZ31513SBR
Component XML file: CP016203.xml
CP016203.xml reports firmware version 1.50
This operation will update the firmware on the
iLO 3 in this server with version 1.50.
Continue (y/N)?y
Current firmware is 1.26 (Aug 26 2011 )
Firmware image is 0x801664(8394340) bytes
Committing to flash part...
******** DO NOT INTERRUPT! ********
Flashing completed.
Attempting to reset device.
Succeeded.
***** iLO 3 reboot in progress (may take up to 60 seconds.)
***** Please ignore console messages, if any.
iLO 3 reboot completed.
Events LOGs - 1 record example showed
hpiLO-> show /map1/log1/record452
status=0
status_tag=COMMAND COMPLETED
Tue Mar 20 15:36:23 2012
/map1/log1/record452
Targets
Properties
number=452
severity=Informational
date=03/20/2012
time=15:22
description=SSH login: root - 192.168.2.21(DNS name not found).
Verbs
cd version exit show set
HTTPS
The simplest mechansim to interact with iLO3 is by starting firefox, like:
firefox rmfs14 &
Remote Virtual Console
You can access the remote video of the server by firefox + Java but obviously the Java plugin wasn't configured inside the firefox installed in
t3admin01
:-), for install it you just need to make a symbolic link like here showed:
[root@t3admin01 plugins]# pwd
/usr/lib64/firefox-3.6/plugins
[root@t3admin01 plugins]# ll
total 0
lrwxrwxrwx 1 root root 46 Feb 1 15:02 libjava.so -> /usr/java/jdk1.6.0_29/jre/lib/amd64/libjava.so
It's worth to mention that
by using the Java Remote Virtual Console you can switch among the several screens that Linux provides, basically you'll find the equivalent of
ALT+F[1-8]
, those are very useful during a Linux installation.