KeyWords: (see the KeyWords page for guidelines on what to put here.)

Comparison between Xen, VMWare and KVM

Disk performance

Xen

The test is done over /root, where the disk is a logical volume specially for the virtual machine. Raid1 over two disks is behind.

Dec 06 13:34 [root@pub:~]# dd if=/dev/zero of=/root/thing bs=1024 count=1024k
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 12.9547 seconds, 82.9 MB/s

Dec 06 11:55 [root@pub:~]# bonnie++ -u root -s 7000 -r 0 -b -d /root/
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
pub.lcg.cscs. 7000M   451  98 53494  11 29537   1   807  66 94265   2 257.5   0
Latency             23957us    2261ms    2977ms     648ms    1330ms    1754ms
Version  1.96       ------Sequential Create------ --------Random Create--------
pub.lcg.cscs.ch     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    55   0 +++++ +++    61   0    65   0 +++++ +++    64   0
Latency              6000ms     499us    3115ms    2426ms      15us    3305ms

Dec 06 14:08 [root@pub:~]# dd if=/root/thing of=/dev/null
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 9.39562 seconds, 114 MB/s

VMWare

The test is done over /root, where the disk is a file on the host's filesystem. Raid1 over two disks is behind.

[root@ppwn01 ~]# dd if=/dev/zero of=/root/thing bs=1024 count=1024k
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 9.29409 seconds, 116 MB/s

[root@ppwn01 ~]# bonnie++ -u root -s 5000 -r 0 -b -d /root/
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ppwn01.lcg.cs 5000M  1208  91 17624   3 49144   3  3918  95 113098   5 368.3   2
Latency              6520us    6915ms     287ms    8979us   41605us     120ms
Version  1.96       ------Sequential Create------ --------Random Create--------
ppwn01.lcg.cscs.ch  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    80   0 +++++ +++    80   0    80   0 +++++ +++    80   0
Latency               137ms     498us   84625us     149ms      16us   89699us

[root@ppwn01 ~]# dd if=/root/thing of=/dev/null
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 9.67018 seconds, 111 MB/s

KVM

This test is with just one disk behind. With Raid1 it will be much better.

[root@test57 ~]# dd if=/dev/zero of=/root/thing bs=1024 count=1024k
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 35.192 seconds, 30.5 MB/s
[root@test57 ~]# bonnie++ -u root -s 7000 -r 0 -b -d /root/
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
test57.lcg.cs 7000M   900  94 19598   4 29381  11  2105  98 371660  36  1051  70
Latency             11737us    4156ms    2477ms    4485us   17899us     105ms
Version  1.96       ------Sequential Create------ --------Random Create--------
test57.lcg.cscs.ch  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16    53   0 +++++ +++    78   0    53   0 +++++ +++    78   0
Latency               318ms     491us     235ms   72324us     145us     144ms
[root@test57 ~]# dd if=/root/thing of=/dev/null
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 3.61255 seconds, 297 MB/s

Compile kernel

Xen

Dec 06 13:28 [root@pub:~]# time tar -xf linux-2.6.36.1.tar 
real    0m33.963s
user    0m0.124s
sys     0m1.600s

Dec 06 13:39 [root@pub:linux-2.6.36.1]# make defconfig
Dec 06 13:39 [root@pub:linux-2.6.36.1]# time make
[...]
real    10m11.454s
user    6m28.432s
sys     2m12.040s

VMWare

[root@ppwn01 ~]# time tar -xf linux-2.6.36.1.tar 
real    0m14.374s
user    0m0.113s
sys     0m1.707s

[root@ppwn01 linux-2.6.36.1]# make defconfig
[root@ppwn01 linux-2.6.36.1]# time make
[...]
real    9m41.033s
user    6m59.439s
sys     2m39.054s

KVM

[root@test57 ~]# time tar -xf linux-2.6.36.1.tar

real    0m14.459s
user    0m0.171s
sys     0m2.761s
[root@test57 ~]# make defconfig
[root@test57 linux-2.6.36.1]# time make
real    11m46.107s
user    8m21.695s
sys     3m4.949s 

HEPSPEC06

Xen

SPEC2006 results for pub.lcg.cscs.ch: 11.65

VMWare

SPEC2006 results for pub.lcg.cscs.ch: 11.60

KVM

Final result: 11.41

Network

Xen

Dec 07 09:10 [root@pub:~]# iperf -c 148.187.66.232
------------------------------------------------------------
Client connecting to 148.187.66.232, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 148.187.66.50 port 58274 connected with 148.187.66.232 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.11 GBytes    951 Mbits/sec

Sep 02 15:14 [root@pub:~]# /usr/lib/lmbench/bin/x86_64-linux-gnu/bw_tcp xen11
0.065536 86.46 MB/sec
Sep 02 15:17 [root@pub:~]# /usr/lib/lmbench/bin/x86_64-linux-gnu/lat_tcp xen11
TCP latency using xen11: 0.8376 microseconds

VMWare

[root@ppwn01 ~]# ./iperf -c 148.187.70.232
------------------------------------------------------------
Client connecting to 148.187.70.232, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 148.187.68.201 port 51597 connected with 148.187.70.232 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.20 GBytes  1.89 Gbits/sec

KVM

[root@test57 ~]# /usr/lib/lmbench/bin/x86_64-linux-gnu/bw_tcp xen11
0.065536 47.36 MB/sec
[root@test57 ~]# /usr/lib/lmbench/bin/x86_64-linux-gnu/lat_tcp xen11
TCP latency using xen11: 0.5083 microseconds

Conclusions

• CPU power is the same. The differences are within IO
• Network performance is much better in VMWare, reaching up to 4 Gbit/s, because it uses infiniband
• Bulk disk performance is also better in VMWare, but exceeds in small disk operations by at least a factor of 2 in some cases.
• Monitoring in VMWare is also better, there are very nice graphs in the GUI,
• VMWare's GUI needs Windows installed and is the only way to manage/create the virtual machines. Clumsy.

Readers' comments