Tags:
create new tag
view all tags

HP ProLiant DL380 G7 iLO3

Intro

The 2 HP DL380 G7 servers t3fs[13,14] can be managed by the specific HP Service Processor called iLO3 that basically is a dedicated Ethernet port on the rear of the server providing the following management mechanisms:

  • by IPMI
  • by SSH + SL6 cooperation to get the Serial Virtual Console ( it's like to attach a serial cable to the server )
  • by HTTPS + Java/.NET plugins in your browser to get the Remote Virtual Console ( it's like to be in front of the VGA screen of the server, so you can run also graphic commands )
For both server that port is connected to our private 100Mbit/s network 192.168.2.21/24 that's reacheable only through t3admin01, so first you'll need to login into t3admin01 and then interact with iLO3.

You can also configure iLO3 by rebooting the server and entering into the iLO3 BIOS but that's usually a mechanism you won't use because the servers are in production.

Once properly configured and understood you can perform by iLO3 many common Sys Admin operations like:

  • Redirect the console
  • Read HW sensors
  • Apply power on/off/reset cycles
  • Mount a Linux recovery CD/DVD .iso to recover a corrupted installation

Users

When I've mounted in the rack the 2 servers I was asking to myself how to login into iLO3, eventually on the front of the each server there is a (i) flag that you can pull out and read the HP factory id/pwd, but because I didn't notice it I've also created by iLO3 BIOS an user root with the Admin privileges; actually I still prefer this root user I made because it's a common user inside our private network and thus we can forget about the specific iLO3 user.

IPMI

IPMI daemon

First check out if the IPMI daemon is listening on UDP:
[root@t3admin01 sbin]# nmap -sU rmfs13 -p 623

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2013-10-15 16:53 CEST
Interesting ports on rmfs13 (192.168.2.55):
PORT    STATE         SERVICE
623/udp open|filtered unknown
MAC Address: 44:1E:A1:3A:46:B4 (Unknown)

Nmap finished: 1 IP address (1 host up) scanned in 0.310 seconds

ipmitool

Pay attention to that -I lanplus option !
[root@t3admin01 ~]# ipmitool -I lanplus   -H rmfs13  -U root -f /root/private/ipmi-pw sdr
UID Light        | 0 unspecified     | ok
Sys. Health LED  | 0 unspecified     | ok
Power Supply 1   | 65 Watts          | nc
Power Supply 2   | 85 Watts          | nc
Power Supplies   | 0 unspecified     | nc
Fan 1            | 29.40 unspecifi | nc
Fan 2            | 29.40 unspecifi | nc
Fan 3            | 46.26 unspecifi | nc
Fan 4            | 54.49 unspecifi | nc
Fan 5            | 54.49 unspecifi | nc
Fan 6            | 13.72 unspecifi | nc
Fans             | 0 unspecified     | nc
Temp 1           | 19 degrees C      | ok
Temp 2           | 40 degrees C      | ok
Temp 3           | 40 degrees C      | ok
Temp 4           | 31 degrees C      | ok
Temp 5           | 33 degrees C      | ok
Temp 6           | 29 degrees C      | ok
Temp 7           | 33 degrees C      | ok
Temp 8           | 35 degrees C      | ok
Temp 9           | 30 degrees C      | ok
Temp 10          | 40 degrees C      | ok
Temp 11          | 32 degrees C      | ok
Temp 12          | 40 degrees C      | ok
Temp 13          | 27 degrees C      | ok
Temp 14          | 31 degrees C      | ok
Temp 15          | 31 degrees C      | ok
Temp 16          | disabled          | ns
Temp 17          | disabled          | ns
Temp 18          | disabled          | ns
Temp 19          | 25 degrees C      | ok
Temp 20          | 26 degrees C      | ok
Temp 21          | 28 degrees C      | ok
Temp 22          | 28 degrees C      | ok
Temp 23          | 33 degrees C      | ok
Temp 24          | 31 degrees C      | ok
Temp 25          | 27 degrees C      | ok
Temp 26          | 28 degrees C      | ok
Temp 27          | 32 degrees C      | ok
Temp 28          | disabled          | ns
Temp 29          | 35 degrees C      | ok
Temp 30          | 59 degrees C      | ok
Memory           | 0 error           | ok
Power Meter      | 146 Watts         | cr
Cntlr 1 Bay 1    | 0 unspecified     | nc
Cntlr 1 Bay 2    | 0 unspecified     | nc
Cntlr 1 Bay 3    | 0 unspecified     | ok
Cntlr 1 Bay 4    | 0 unspecified     | ok
Cntlr 2 Bay 5    | 0 unspecified     | ok
Cntlr 2 Bay 6    | 0 unspecified     | ok
Cntlr 2 Bay 7    | 0 unspecified     | ok
Cntlr 2 Bay 8    | 0 unspecified     | ok

freeipmi sensors

[root@t3admin01 sbin]#  /usr/sbin/ipmi-sensors -h rmfs13 -D lan_2_0 --config-file /opt/nagios/check_ipmi_sensor.user.pwd.privilege -t Temperature
ID | Name    | Type        | Reading    | Units | Event
14 | Temp 1  | Temperature | 19.00      | C     | 'OK'
15 | Temp 2  | Temperature | 40.00      | C     | 'OK'
16 | Temp 3  | Temperature | 40.00      | C     | 'OK'
17 | Temp 4  | Temperature | 28.00      | C     | 'OK'
18 | Temp 5  | Temperature | 30.00      | C     | 'OK'
19 | Temp 6  | Temperature | 29.00      | C     | 'OK'
20 | Temp 7  | Temperature | 32.00      | C     | 'OK'
21 | Temp 8  | Temperature | 35.00      | C     | 'OK'
22 | Temp 9  | Temperature | 29.00      | C     | 'OK'
23 | Temp 10 | Temperature | 40.00      | C     | 'OK'
24 | Temp 11 | Temperature | 31.00      | C     | 'OK'
25 | Temp 12 | Temperature | 39.00      | C     | 'OK'
26 | Temp 13 | Temperature | 26.00      | C     | 'OK'
27 | Temp 14 | Temperature | 30.00      | C     | 'OK'
28 | Temp 15 | Temperature | 30.00      | C     | 'OK'
29 | Temp 16 | Temperature | N/A        | C     | N/A
30 | Temp 17 | Temperature | N/A        | C     | N/A
31 | Temp 18 | Temperature | N/A        | C     | N/A
32 | Temp 19 | Temperature | 25.00      | C     | 'OK'
33 | Temp 20 | Temperature | 26.00      | C     | 'OK'
34 | Temp 21 | Temperature | 28.00      | C     | 'OK'
35 | Temp 22 | Temperature | 27.00      | C     | 'OK'
36 | Temp 23 | Temperature | 32.00      | C     | 'OK'
37 | Temp 24 | Temperature | 31.00      | C     | 'OK'
38 | Temp 25 | Temperature | 27.00      | C     | 'OK'
39 | Temp 26 | Temperature | 28.00      | C     | 'OK'
40 | Temp 27 | Temperature | 31.00      | C     | 'OK'
41 | Temp 28 | Temperature | N/A        | C     | N/A
42 | Temp 29 | Temperature | 35.00      | C     | 'OK'
43 | Temp 30 | Temperature | 58.00      | C     | 'OK'

Nagios check based on freeipmi sensors

[root@t3admin01 sbin]# /opt/nagios/check_ipmi_sensor -O '-D lan_2_0 -t Temperature' -f /opt/nagios/check_ipmi_sensor.user.pwd.privilege -H rmfs13
IPMI Status: OK | 'Temp 1'=19.00 'Temp 2'=40.00 'Temp 3'=40.00 'Temp 4'=28.00 'Temp 5'=30.00 'Temp 6'=29.00 'Temp 7'=32.00 'Temp 8'=35.00 'Temp 9'=29.00 'Temp 10'=40.00 'Temp 11'=31.00 'Temp 12'=39.00 'Temp 13'=26.00 'Temp 14'=30.00 'Temp 15'=30.00 'Temp 19'=25.00 'Temp 20'=26.00 'Temp 21'=28.00 'Temp 22'=27.00 'Temp 23'=32.00 'Temp 24'=31.00 'Temp 25'=27.00 'Temp 26'=28.00 'Temp 27'=31.00 'Temp 29'=35.00 'Temp 30'=58.00

SSH

Be aware that you can even install your daily SSH PUB key into ILO, this will simplify your logins:
[root@t3admin01 ~]# ssh rmfs13
User:root logged-in to ILOCZ31513SBR.(192.168.2.55)
iLO 3 Advanced 1.26 at  Aug 26 2011
Server Name: t3fs13
Server Power: On

hpiLO-> show
status=0
status_tag=COMMAND COMPLETED
Tue Feb 14 11:19:54 2012

/
  Targets
    system1
    map1
  Properties
  Verbs
    cd version exit show

Basically the /system1 directory is an abstraction to read/manage the server parameters like for instance request a server reboot, read sensors, and so on while the directory /map1 models the specific iLO3 parameters, for instance you might reset the iLO3 module itself: the word Verbs lists the actions you can perform inside a directory. The simplest action is to jump into an other directory and list the available commands for that directory:

hpiLO-> cd /map1

/map1

hpiLO-> show

/map1
  Targets
    firmware1
    accounts1
    log1
    enetport1
    dhcpendpt1
    dnsendpt1
    gateway1
    dnsserver1
    dnsserver2
    dnsserver3
    dhcpserver1
    settings1
    config1
    snmp1
    oemhp_dircfg1
    oemhp_vm1
    vlan1
    oemhp_ssocfg1
  Properties
    name=iLO 3 Advanced
    license=35SCT5KKSJ5GXVDZM75KPW8ZH
  Verbs
    cd version exit show reset set oemhp_ping

hpiLO-> show config1

/map1/config1
  Targets
  Properties
    oemhp_mapenable=yes
    oemhp_timeout=30
    oemhp_rbsuenable=yes
    oemhp_rbsulogin=no
    oemhp_rbsushowip=yes
    oemhp_httpport=80
    oemhp_sslport=443
    oemhp_rcport=17990
    oemhp_vmport=17988
    oemhp_sshport=22
    oemhp_sshstatus=yes
    oemhp_serialclistatus=yes
    oemhp_serialcliauth=no
    oemhp_serialclispeed=115200
    oemhp_minpwdlen=8
    oemhp_enforce_aes=no
    oemhp_authfailurelogging=3
    oemhp_computer_lock=disabled
  Verbs
    cd version exit show set oemhp_loadSSHkey
hpiLO->

Serial Virtual Console

SL6 cooperation

One of the most useful commands you get from iLO3 is the serial redirection of the Linux console like in this example:
hpiLO-> vsp

Virtual Serial Port Active: COM2

Starting virtual serial port.
Press 'ESC (' to return to the CLI Session.

Scientific Linux release 6.0 (Carbon)
Kernel 2.6.32-220.4.1.el6.x86_64 on an x86_64

t3fs13.psi.ch login:

Like adviced, by pressing the keys combinations ESC + SHIFT + ( you will exit from the Serial Virtual Console; please notice that COM2 maps in Linux to ttyS1 and it's explicitely reported in these 2 files:

[root@t3fs14 sdb]# grep S1 /etc/securetty 
ttyS1
[root@t3fs14 sdb]# grep S1 /boot/grub/menu.lst 
       kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 ro root=UUID=a247aed2-7b16-4306-9485-2adc3f62a6da  rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=ttyS1,115200 crashkernel=auto elevator=deadline

Indeed since SL6 it's no more needed to write lines like this:

# grep S1 /etc/inittab
T0:2345:respawn:/sbin/agetty -L -i 115200 ttyS1 vt102

to redirect the console; SL6 will take into account the grub parameter console and it will activate the permanent redirection.

By using the Serial Virtual Console you'll be able to see even the HW initialization phase when you'll reboot these servers and later the GRUB dialog.

SL6 installation

The Serial Virtual Console is also useful during an SL6 installation but again you need the proper cooperation from SL6; to achieve that I've modified our Kickstart installation parameters like showed:

[root@t3admin01 pxelinux.cfg]# cat default
default localboot
prompt 1
timeout 300
display boot.msg
F1 boot.msg

################################
#  Local Boot (default)
################################

label localboot
  LOCALBOOT 0

######################################
#  Scientific Linux 6  2012-02 by martinelli
######################################
label sl60
  kernel scientific/6x/x86_64/vmlinuz
  append text initrd=scientific/6x/x86_64/initrd.img ks=http://linux.web.psi.ch/kickstart/configs/sl60-64-tier3-ks.cfg noipv6 console=ttyS1,115200 ksdevice=eth2 cmdline
0
...

Please pay attention to those append parameters also described inside the official installation guide:

  • ksdevice : is used to explicitly cite the Ethernet port to be used during the Kickstart installation, so Anaconda can skip the question about that ( without that option Anaconda was stopping and asking )
  • cmdline : When cmdline is specified, output on line-mode terminals (such as 3270 under z/VM or operating system messages for LPAR) becomes readable, as the installer disables escape terminal sequences that are only applicable to UNIX-like consoles. This requires installation with a kickstart file that answers all questions, since the installer does not support interactive user input in cmdline mode.

Firmware version

hpiLO-> show /map1/firmware1
status=0
status_tag=COMMAND COMPLETED
Tue Mar 20 15:32:26 2012
/map1/firmware1
  Targets
  Properties
    version=1.26
    date=Aug 26 2011
  Verbs
    cd version exit show load set

Firmware update

made on 27th Dec '13
[root@t3fs13 CP016203]# ./flash_ilo3 

FLASH_iLO3 v1.08 for Linux (May  8 2012)
(C) Copyright 2002-2012 Hewlett-Packard Development Company, L.P.
Firmware image: ilo3_150.bin
Current iLO 3 firmware version  1.26; Serial number ILOCZ31513SBR      

Component XML file: CP016203.xml
CP016203.xml reports firmware version 1.50
This operation will update the firmware on the
iLO 3 in this server with version 1.50.
Continue (y/N)?y
Current firmware is 1.26 (Aug 26 2011 )
Firmware image is 0x801664(8394340) bytes
Committing to flash part...        
******** DO NOT INTERRUPT! ********
Flashing completed.                                   
Attempting to reset device.      
Succeeded.
***** iLO 3 reboot in progress (may take up to 60 seconds.)
***** Please ignore console messages, if any.

iLO 3 reboot completed.

Events LOGs - 1 record example showed

hpiLO-> show /map1/log1/record452
status=0
status_tag=COMMAND COMPLETED
Tue Mar 20 15:36:23 2012
/map1/log1/record452
  Targets
  Properties
    number=452
    severity=Informational
    date=03/20/2012
    time=15:22
    description=SSH login: root - 192.168.2.21(DNS name not found).
  Verbs
    cd version exit show set

HTTPS

The simplest mechansim to interact with iLO3 is by starting firefox, like:

firefox rmfs14 &

Remote Virtual Console

You can access the remote video of the server by firefox + Java but obviously the Java plugin wasn't configured inside the firefox installed in t3admin01 :-), for install it you just need to make a symbolic link like here showed:

[root@t3admin01 plugins]# pwd
/usr/lib64/firefox-3.6/plugins
[root@t3admin01 plugins]# ll
total 0
lrwxrwxrwx 1 root root 46 Feb  1 15:02 libjava.so -> /usr/java/jdk1.6.0_29/jre/lib/amd64/libjava.so

It's worth to mention that by using the Java Remote Virtual Console you can switch among the several screens that Linux provides, basically you'll find the equivalent of ALT+F[1-8], those are very useful during a Linux installation.

dCache role of these servers

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r9 - 2016-11-04 - FabioMartinelli
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback