Phoenix Hardware and OS related services description
Here are described the infrastructure services run on, together with procedures related to them, drivers, incompatibilities, monitoring, testing and other useful information. We will have two sections: Hardware Cards and OS Features.
Hardware Cards
Here is a list of the Hardware Cards we have filled in so far.
You can add new ones in HardwareCards.
Other Hardware or OS related documents
Hardware control.
The ILOM network is in a separate network (192.168.*). You can access it through the Xen hosts (i.e. Xen12) and perform ON/OFF commands with
ion
,
ioff
or
ireset
VNIC setup
To set the VNIC on any machine, we need to install all the mlx4 packages available from the MLNX ISO and configure a few files:
- If the kernel and/or the SL version, download the right ISO file from the Mellanox website (http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=26&menu_section=34#tab-three).
- Do the following steps:
mount -o loop ./MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso /mnt
cp -r /mnt/ ./mlnx
# MAKE SURE .mlnx is also copied
umount /mnt
cd mlnx/docs
vim mlnx_add_kernel_support.sh
# add this so the script detects SL as redhat:
# sl-release-*6.0*|
./mlnx_add_kernel_support.sh --iso /root/MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso
# Once it is finished, the new image is created in /tmp, replace the one we downloaded with it:
mv /tmp/MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso /root/
mount -o loop ./MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso /mnt
/mnt/mlnxofedinstall
# Done!
- Once these steps are done, reboot the machine.
- If the package
infiniband-diags
is not installed, install it from the iso image just created. This package provides utilities that we can use to test the status of the infiniband fabric. In particular, ibstat
, ibstatus
, ibhosts
, ibswitches
and ibping
are really useful.
- At this point, the infiniband card should be up and running if the
ifcfg-ib0
file is correct and the fabric is properly connected.
- To install the VNIC, modify
/etc/infiniband/openib.conf
sed -i -e 's/MLX4_VNIC_LOAD=no/MLX4_VNIC_LOAD=yes/' /etc/infiniband/openib.conf
And create /etc/infiniband/mlx4_vnic.conf
as follows (in this example, eth0 has the 10.10 IP): VMAC1=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $1}'`
VMAC2=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $2}'`
VMAC3=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $3}'`
VMAC4=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $4}'`
VMAC5=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $5}'`
VMAC6=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $6}'`
IP=`/sbin/ifconfig eth0 | grep 'inet addr:' | awk ' { print $2 } ' | cut -d. -f4-`
SN=`expr $IP + 100`
echo "name=eth2 mac=00:00:${VMAC3}:${VMAC4}:${VMAC5}:${VMAC6} ib_port=mlx4_0:1 vnic_id=${SN} bx=00:02:C9:03:00:11:11:77 eport=A10" > /etc/infiniband/mlx4_vnic.conf
- Then, create the proper file
ifcfg-eth2
and the VNIC should be working well.
Shared filesystems
This page's children (to organize)
--
PabloFernandez - 2011-01-20