Tags:
tag this topic
create new tag
view all tags
<!-- keep this as a security measure: * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup #uncomment this if you want the page only be viewable by the internal people #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup --> ---+ Mellanox Yum install %TOC% ---++ Create the repo The Mellanox can now easily be installed via a yum repo. Instructions can be found in the user guide http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v2.0-3.0.0.pdf Unpack the tarball and place in with you other repos <verbatim> tar xzvf MLNX_OFED_LINUX-2.0-3.0.0-rhel6.4-x86_64.tgz mkdir /var/www/html/mlnx mv MLNX_OFED_LINUX-2.0-3.0.0-rhel6.4-x86_64/* /var/www/html/mlnx </verbatim> There is already repodata directory provided so we don't have to run createrepo. ---++ Add the repo to clients On the nodes we want to install we need to add the following repo. Mellanox also sign the packages so we can use keys. Note the priority used is "1", the priority of this repo must be higher (represented with a lower number) than the repo for the SL packages. Otherwise packages like libverbs will be pulled from SL rather than the Mellanox repo. <verbatim> # This should be done with cfengine for site wide deployment cd /etc/pki/rpm-gpg/ wget http://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox vim /etc/yum.repos.d/mlnx.repo [mlnx] name=Phoenix Mellanox repo enabled=1 gpgcheck=1 baseurl=http://phoenix1.lcg.cscs.ch:81/mlnx/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Mellanox # To use priorities you must have yum-priorities installed priority=1 </verbatim> ---++ Install the packages Firstly remove and SL provided OFED packages <verbatim> yum groupremove Infiniband\ Support </verbatim> Install the Mellanox stack from the repo. Check the repo name listed and ensure all the packages are from the Mellanox repo. Also make sure you haven't ssh'd in via the IB interface as this will go down during the install. <verbatim> yum groupinstall 'MLNX_OFED ALL' --disableexcludes=main cfagent -q </verbatim> Then reboot the machine. ---+++ Installing on a RHEL node previously installed with RHEL stack There are a few small variations done in order to get this done on a RHEL 6.x machine that was previously installed with the standard kernel stack: <verbatim> -- /etc/yum.repos.d/mlnx -- [mlnx] name=Phoenix Mellanox repo enabled=1 baseurl=http://phoenix1.lcg.cscs.ch:81/mlnx/6.5 # To use priorities you must have yum-priorities installed priority=1 </verbatim> <verbatim>yum groupinstall 'MLNX_OFED ALL' --disablerepos=* --enablerepo=mlnx </verbatim> ---+++ MTU note I noticed after rebooting the machine the MTU was set to 2044 despite having connected mode enabled in openib.conf It appears the interface has been loaded in datagram mode <verbatim> cat /sys/class/net/ib0/mode datagram </verbatim> Looking into the RPMs there are scripts that echo into obenib.conf for some stupid reason. As such run cfengine and reboot if you have this issue. A "service openibd restart" will often not be possible as there will be many things using the IB interface e.g. GPFS and NFS mounts, easier just to reboot. ---++ Kernel dependencies It appears Mellanox has altered their packaging method to something more sane. There is no longer a kernel-ib package which is strictly bound to one kernel version. Instead there are kmod packages which provide modules and execute depmod as part of their RPM scripts. Case in point I installed the Mellanox RPMs on a newly installed machine running the latest SL 6.4 kernel 2.6.32-358.23.2 without any issue. Need to test kernel upgrades to see if this breaks anything. 04/12/2013 GB - Tested installing on an older kernel and updating said kernel, Mellanox stack seemed fine after kernel update -- Main.GeorgeBrown - 2013-11-18
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r7
<
r6
<
r5
<
r4
<
r3
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r7 - 2014-05-08
-
MiguelGila
LCGTier2
Log In
(Topic)
LCGTier2 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Users
Entry point / Contact
RoadMap
ATLAS Pages
CMS Pages
CMS User Howto
CHIPP CB
Outreach
Technical
Cluster details
Services
Hardware and OS
Tools & Tips
Monitoring
Logs
Maintenances
Meetings
Tests
Issues
Blog
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
LCGTier2 Web
Users
Groups
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Warning: Can't find topic "".""
Account
Log In
E
dit
A
ttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback