Mellanox Yum install

Create the repo

The Mellanox can now easily be installed via a yum repo. Instructions can be found in the user guide http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v2.0-3.0.0.pdf

Unpack the tarball and place in with you other repos

tar xzvf MLNX_OFED_LINUX-2.0-3.0.0-rhel6.4-x86_64.tgz

mkdir /var/www/html/mlnx

mv MLNX_OFED_LINUX-2.0-3.0.0-rhel6.4-x86_64/* /var/www/html/mlnx

There is already repodata directory provided so we don't have to run createrepo.

Add the repo to clients

On the nodes we want to install we need to add the following repo. Mellanox also sign the packages so we can use keys.

Note the priority used is "1", the priority of this repo must be higher (represented with a lower number) than the repo for the SL packages. Otherwise packages like libverbs will be pulled from SL rather than the Mellanox repo.

# This should be done with cfengine for site wide deployment
cd /etc/pki/rpm-gpg/
wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox

vim /etc/yum.repos.d/mlnx.repo

  [mlnx]
  name=Phoenix Mellanox repo
  enabled=1
  gpgcheck=1
  baseurl=http://phoenix1.lcg.cscs.ch:81/mlnx/
  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Mellanox
  # To use priorities you must have yum-priorities installed
  priority=1

Install the packages

Firstly remove and SL provided OFED packages

yum groupremove Infiniband\ Support

Install the Mellanox stack from the repo. Check the repo name listed and ensure all the packages are from the Mellanox repo. Also make sure you haven't ssh'd in via the IB interface as this will go down during the install.

yum groupinstall 'MLNX_OFED ALL' --disableexcludes=main
cfagent -q

Then reboot the machine.

Installing on a RHEL node previously installed with RHEL stack

There are a few small variations done in order to get this done on a RHEL 6.x machine that was previously installed with the standard kernel stack:

 -- /etc/yum.repos.d/mlnx --
  [mlnx]
  name=Phoenix Mellanox repo
  enabled=1
  baseurl=http://phoenix1.lcg.cscs.ch:81/mlnx/6.5
  # To use priorities you must have yum-priorities installed
  priority=1

yum groupinstall 'MLNX_OFED ALL' --disablerepos=* --enablerepo=mlnx

MTU note

I noticed after rebooting the machine the MTU was set to 2044 despite having connected mode enabled in openib.conf

It appears the interface has been loaded in datagram mode

cat /sys/class/net/ib0/mode                                          
datagram

Looking into the RPMs there are scripts that echo into obenib.conf for some stupid reason. As such run cfengine and reboot if you have this issue. A "service openibd restart" will often not be possible as there will be many things using the IB interface e.g. GPFS and NFS mounts, easier just to reboot.

Kernel dependencies

It appears Mellanox has altered their packaging method to something more sane. There is no longer a kernel-ib package which is strictly bound to one kernel version. Instead there are kmod packages which provide modules and execute depmod as part of their RPM scripts. Case in point I installed the Mellanox RPMs on a newly installed machine running the latest SL 6.4 kernel 2.6.32-358.23.2 without any issue. Need to test kernel upgrades to see if this breaks anything.

04/12/2013 GB - Tested installing on an older kernel and updating said kernel, Mellanox stack seemed fine after kernel update

-- GeorgeBrown - 2013-11-18