Phoenix Hardware and OS related services description

Here are described the infrastructure services run on, together with procedures related to them, drivers, incompatibilities, monitoring, testing and other useful information. We will have two sections: Hardware Cards and OS Features.

Hardware Cards

Here is a list of the Hardware Cards we have filled in so far. You can add new ones in HardwareCards.

Page Name Used for N. in Production OS Inside First puchase CPU (HS06) Disk (MB/s) Power (Watts) Last modified
HardwareE5500B NetApp E5500B GPFS2 2 No Feb 2014       2014-02-12 - 10:32
HardwareDS3700 IBM DCS3700 dCache 6 No 12.2012       2013-02-14 - 12:32
HardwareThor Sun X4540 dCache 10 Yes Dec09   300 1000 2012-07-05 - 09:18
HardwareStorageElement                 2011-04-18 - 09:38
HardwareSunBlades6048 Sun SB6048 WNs 4 Yes Dec 2009 2825 N/A 3750-6250 2011-06-10 - 08:51
HardwareThumper Sun X4500 dCache storage 29 Yes Dec 2008 N/A 400 1130 2011-07-13 - 06:57
HardwareDS3500 IBM DS3500 dCache and GPFS storage 3 No Dec 2010 N/A 1500 ? 2011-03-07 - 10:40

Other Hardware or OS related documents

Hardware control.

The ILOM network is in a separate network (192.168.*). You can access it through the Xen hosts (i.e. Xen12) and perform ON/OFF commands with ion, ioff or ireset

VNIC setup

To set the VNIC on any machine, we need to install all the mlx4 packages available from the MLNX ISO and configure a few files:

  1. If the kernel and/or the SL version, download the right ISO file from the Mellanox website (http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=26&menu_section=34#tab-three).
  2. Do the following steps:
     mount -o loop ./MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso  /mnt
    cp -r /mnt/ ./mlnx 
    # MAKE SURE .mlnx is also copied
    umount /mnt
    cd mlnx/docs
    vim mlnx_add_kernel_support.sh
    # add this so the script detects SL as redhat: 
    # sl-release-*6.0*|
    ./mlnx_add_kernel_support.sh --iso /root/MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso
    # Once it is finished, the new image is created in /tmp, replace the one we downloaded with it:
    mv /tmp/MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso /root/
    mount -o loop ./MLNX_OFED_LINUX-1.5.3-3.0.0-rhel6-x86_64.iso  /mnt
    /mnt/mlnxofedinstall
    # Done! 
  3. Once these steps are done, reboot the machine.
  4. If the package infiniband-diags is not installed, install it from the iso image just created. This package provides utilities that we can use to test the status of the infiniband fabric. In particular, ibstat, ibstatus, ibhosts, ibswitches and ibping are really useful.
  5. At this point, the infiniband card should be up and running if the ifcfg-ib0 file is correct and the fabric is properly connected.
  6. To install the VNIC, modify /etc/infiniband/openib.conf
    sed -i -e 's/MLX4_VNIC_LOAD=no/MLX4_VNIC_LOAD=yes/' /etc/infiniband/openib.conf
    And create /etc/infiniband/mlx4_vnic.conf as follows (in this example, eth0 has the 10.10 IP):
    VMAC1=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $1}'`
    VMAC2=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $2}'`
    VMAC3=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $3}'`
    VMAC4=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $4}'`
    VMAC5=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $5}'`
    VMAC6=`ifconfig |grep eth0|awk '{print $5}'|awk --field-separator ":" '{print $6}'`
    
    IP=`/sbin/ifconfig eth0 | grep 'inet addr:' | awk ' { print $2 } ' | cut -d. -f4-`
    SN=`expr $IP + 100`
    
    echo "name=eth2 mac=00:00:${VMAC3}:${VMAC4}:${VMAC5}:${VMAC6} ib_port=mlx4_0:1 vnic_id=${SN} bx=00:02:C9:03:00:11:11:77 eport=A10" > /etc/infiniband/mlx4_vnic.conf
  7. Then, create the proper file ifcfg-eth2 and the VNIC should be working well.

Shared filesystems

This page's children (to organize)

PageSorted ascending Title Date created Date last modified
BenchMarks 2010-02-18 - 09:36 2016-02-11 - 14:08
EnableSendingMails 2010-05-26 - 07:49 2011-02-14 - 08:54
HardwareCards Hardware Cards 2011-01-20 - 17:49 2011-02-28 - 15:27
HardwareInventory 2010-02-11 - 18:55 2011-02-14 - 09:13
IBMStorageNotes 2010-11-25 - 16:20 2011-01-21 - 15:51
LustreDiskChangeProcedures 2010-05-18 - 08:32 2011-06-24 - 09:39
LustreFreezeUpProcedures 2010-07-26 - 09:14 2011-05-31 - 06:46
LustreInformation 2010-04-13 - 07:42 2011-08-12 - 09:50
SL5UpgradePlanning Discussion about the upgrade of WNs to SL(C)5 2009-06-18 - 10:03 2011-01-21 - 15:16
WorkingWithVMWareESXi4 Working with VMWare ESXi 4 2010-12-02 - 15:09 2011-01-24 - 13:19
XenGuestOsInstallation Installing in a PV (para-virtualized) machine 2010-02-15 - 16:53 2011-08-08 - 14:56
XenSampleImageReplication Creation of a Xen Sample image 2010-02-18 - 10:48 2011-09-02 - 09:35

-- PabloFernandez - 2011-01-20

Edit | Attach | Watch | Print version | History: r14 < r13 < r12 < r11 < r10 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r11 - 2012-01-17 - MiguelGila
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback