create new tag
view all tags

Keywords: vmware, esx, esxi, console, ssh, infiniband

Working with VMWare ESXi 4

Setting up a VMWare ESXi 4 host

  • Installation with the ILOM
    • in the ILOM, redirect the installation cdrom (/extras/VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso) and reboot the machine icdrom xx, ireset xx
    • Follow the installation, it's just continue-continue.
    • When you see the welcome screen press F2 to customize it.
    • The first password is empty. Set it in the menu.
    • Check that the management network is correct (it should be in the 10.10 network, taken from DHCP)
    • Enter also Troubleshooting Options and enable Remote Tech Support (SSH)
    • You can exit the ILOM, the basic setup is done.
    • Keep in mind the security if you assign a public IP to the admin interface. There is more than just the ssh interface.
  • Copy the needed files into the ESXi host (drivers and script wrapper, check attachments) from *xen02*
    scp /extra/MEL-OFED-1.4.1-375-offline_bundle.zip
    scp /extra/vm
  • Log in the ESXi Host to make some changes
  • Enable IPoIB Infiniband support drivers
    ssxupdate --bundle /scratch/MEL-OFED-1.4.1-375-offline_bundle.zip update
  • Activate the license with this (see VmwareEsxiLicense)
    vim-cmd vimsvc/license --set XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
  • Disallow passord login
    sed 's/shell -i/shell -i -s/' -i /etc/inetd.conf
  • Put some private keys to be able to ssh into the machine
    echo '# Enable pubkey login
    mkdir /.ssh
    chmod 700 /.ssh
    echo "ssh-dss XXX
    " > /.ssh/authorized_keys
    ' > /etc/rc.local 
    vi /etc/rc.local
  • Enable auto-start and soft-shutdown of guests by default
    vim-cmd hostsvc/autostartmanager/enable_autostart 1
    vim-cmd hostsvc/autostartmanager/update_defaults 120 120 GuestShutdown 0
  • Backup the actual config to make it work in the next bootup. Do it twice! (does not copy everything in the first run)
  • Check rc.local and inetd.conf were backed up. Then Reboot the machine, and log in again

Configure infiniband networking

  • Check the actual network configuration in the host
    esxcfg-nics -l
  • Enable IPoIB in both infiniband ports
    esxcfg-module -s "port_types=24,1,1,24,1,1 port_type_default=1" mlx4_en
  • Verify that both ports are visible
    esxcfg-nics -l
  • Create a virtual Switch with one of the ports, for all the virtual machines to use it (involves creating a vSwitch, linking a device to it, and create a PortGroup with it)
    esxcfg-vswitch -a vSwitch1
    esxcfg-vswitch -L vmnic18.p1 vSwitch1
    esxcfg-vswitch -A ib0 vSwitch1
    esxcfg-vswitch -l

At the end of this process, there will be a PortGroup called ib0 where we can plug in our virtual machines. Everything should be ready now, you can proceed creating guests!

Operations with VM Guests


  • In the Host, go to
    cd /vmfs/volumes/datastore1
  • There you should see all VM Guest filesystems, check also the available size
    df -h| egrep "vmfs3|Used"
  • Create your own directory
    mkdir $VM
    cd $VM
  • Create a virtual disk
    vmkfstools -c 12G -a lsilogic $VM.vmdk
  • Create the VM definition file with vi $VM.vmx and put inside
    config.version = "8"
    virtualHW.version= "7"
    guestOS = "rhel5-64"
    memsize = "1024"
    displayname = "@VM@"
    scsi0.present = "TRUE"
    scsi0.virtualDev = "lsilogic"
    scsi0:0.fileName = "@VM@.vmdk"
    scsi0:0.present = "TRUE"            
    ethernet0.address = "@MAC@"                
    ethernet0.addressType = "static"                       
    ethernet0.networkName = "VM Network"                   
    ethernet0.present = "TRUE"                             
    ethernet0.virtualDev = "e1000"                         
    ethernet1.addressType = "generated"    
    ethernet1.networkName = "ib0"          
    ethernet1.present = "TRUE"             
    ethernet1.virtualDev = "vmxnet3"
  • Replace the values inside
    sed "s/@VM@/$VM/" -i $VM.vmx
    sed "s/@MAC@/00:50:56:68:02:02/" -i $VM.vmx
    chmod 744 $VM.vmx
  • Activate the VM (make it visible for ESXi) and vereify it
    vim-cmd solo/registervm /vmfs/volumes/datastore1/$VM/$VM.vmx $VM
    vim-cmd vmsvc/getallvms
  • Later on you should install the vmware tools from inside the guest # /root/vmware-install.pl --default

Power control

  • Check state
    vim-cmd vmsvc/power.getstate 16
  • Power on
    vim-cmd vmsvc/power.on 16
  • Shutdown
    vim-cmd vmsvc/power.shutdown 16
  • Hard stop
    vim-cmd vmsvc/power.off 16
  • Soft reboot
    vim-cmd vmsvc/power.reboot 16
  • Reset
    vim-cmd vmsvc/power.reset 16

Other operations

  • List available guests
    vim-cmd vmsvc/getallvms
  • Get extra online information from VM
    vim-cmd vmsvc/get.summary 16
  • Install VMWare Tools (don't use it here, we do it in another way)
    vim-cmd vmsvc/tools.install 16
  • Reload changes done in the .vmx file
    vim-cmd vmsvc/reload 16
  • Unregister a machine (run before removing the files!)
    vim-cmd vmsvc/unregister 16
  • Check running machines CPU usage
  • See the system console (what you see from the ILOM)

Working with the GUI

Some times you would like to work with the User Interface (it's called vSphere Client)
  • Identify the ID of the windows VM guest on xen02 (with xm list. In this example, id=11)
  • Open a tunnel from your workstation to route VNC traffic ssh root@xen02 -L 12345:localhost:5911 (change the 11 for the ID from before)
  • Run in your desktop vncviewer localhost:12345
  • Start the VMWare vSphere Client, and open the session to the 10.10 management interface of the host.

Working with the console

You need to keep in mind that the root filesystem is overwritten every time you reboot. To make changes permanent you need to edit an already existing file in /etc (for example) and run /sbin/auto-backup.sh. If you want to add actions to the boot procedure, edit the /etc/rc.local and do it permanent.

Also, you can use /sbin/services.sh restart to restart all enabled services, specified in /etc/chkconfig.db. This does not affect the virtual machine guests.

External links

There are some pages on how to manage ESXi from the command line:

And maybe, at some point, you want to use this, but probably not: http://www.vmware.com/support/developer/vima/

-- PabloFernandez - 2010-12-02

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r9 - 2011-01-24 - PabloFernandez
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback