Lustre Oss hardware

Using 3 Virident cards, 2 as metadataOnly, 1 as dataAndMetadata - 256K blocksize

Sep 14 11:19 [root@wn120:tmpgpfs]# dd if=/dev/zero of=test bs=4096k count=10000
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB) copied, 37.7524 seconds, 1.1 GB/s

Sep 14 11:20 [root@wn120:tmpgpfs]# time tar xf linux-2.6.36.1.tar 
real    0m13.559s
user    0m0.267s
sys     0m3.727s

make
real    8m12.997s
user    6m56.126s
sys     0m58.973s

Using 3 virident cards as metadata + 3 raids, 2 on oss11, 1 on oss12 - 256K blocksize

Sep 14 12:01 [root@wn120:tmpgpfs]# dd if=/dev/zero of=test bs=4096k count=10000
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB) copied, 232.677 seconds, 180 MB/s

Sep 14 12:27 [root@wn120:tmpgpfs]# time tar xf linux-2.6.36.1.tar 
real    0m37.709s
user    0m0.051s
sys     0m0.773s

time make
real    8m26.329s
user    6m56.563s
sys     0m58.123s

dd test on three hosts
Sep 14 12:42 [root@xen11:~]# dsh -w wn[120-122] /tmpgpfs/testgpfs.sh
wn120: 10000+0 records in
wn120: 10000+0 records out
wn120: 41943040000 bytes (42 GB) copied, 666.36 seconds, 62.9 MB/s
wn120: 
wn120: real     11m6.362s
wn120: user     0m0.025s
wn120: sys      0m20.165s
wn121: 10000+0 records in
wn121: 10000+0 records out
wn121: 41943040000 bytes (42 GB) copied, 666.798 seconds, 62.9 MB/s
wn121: 
wn121: real     11m6.800s
wn121: user     0m0.011s
wn121: sys      0m18.103s
wn122: 10000+0 records in
wn122: 10000+0 records out
wn122: 41943040000 bytes (42 GB) copied, 671.016 seconds, 62.5 MB/s
wn122: 
wn122: real     11m11.017s
wn122: user     0m0.011s
wn122: sys      0m17.614s

on ten hosts, it's about 10 MB/s

Xen16 machine

dd

[root@xen16 ~]# dd if=/dev/zero of=/virident/thing bs=8k count=1024k
1048576+0 records in
1048576+0 records out
8589934592 bytes (8.6 GB) copied, 13.6994 s, 627 MB/s

[...] do some stuff

[root@xen16 ~]# dd if=/virident/thing of=/dev/null
^[[6~16777216+0 records in
16777216+0 records out
8589934592 bytes (8.6 GB) copied, 18.8337 s, 456 MB/s

Bonnie

[root@xen16 ~]# bonnie++ -u root -s 60000 -r 0 -b -d /virident/
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xen16.lcg.cs 60000M   733  99 530557  94 298348  78  3522  99 722689  88  1078 112
Latency             11048us   14500us     194ms    2686us    4304us    5686us
Version  1.96       ------Sequential Create------ --------Random Create--------
xen16.lcg.cscs.ch   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1599   6 +++++ +++  2253   7  1777   8 +++++ +++  2248   6
Latency               557ms     375us    4239us    6312us      17us    5872us

Kernel compile

[root@xen16 virident]# time tar -xf linux-2.6.36.1.tar

real    0m1.828s
user    0m0.123s
sys     0m1.706s

[root@xen16 linux-2.6.36.1]# time make
real    7m53.340s
user    6m46.717s
sys     1m18.671s

KVM over local Virident card (xen16)

dd

[root@xen16 ~]# dd if=/dev/zero of=/root/thing bs=4k count=1024k
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 25.7644 seconds, 167 MB/s
[...] do some stuff

[root@xen16 ~]# dd if=/root/thing of=/dev/null
8388608+0 records in
8388608+0 records out
4294967296 bytes (4.3 GB) copied, 12.177 seconds, 353 MB/s

Bonnie

[root@xen16 ~]# bonnie++ -u root -s 6000 -r 0 -b -d /root/
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ppwn04.lcg.cs 6000M   870  99 139143  47 93778  37  3428  97 395061  38  3353 238
Latency             10000us    1418ms     828ms    3999us   10999us   73999us
Version  1.96       ------Sequential Create------ --------Random Create--------
ppwn04.lcg.cscs.ch  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   905   3 +++++ +++   890   2   888   4 +++++ +++   892   2
Latency             30999us    1000us   21000us   32999us    1000us   16000us

Kernel compile

[root@xen16 virident]# time tar -xf linux-2.6.36.1.tar
real    0m3.740s
user    0m0.174s
sys     0m2.973s


[root@xen16 linux-2.6.36.1]# time make

KVM over remote Virident card mounted on NFS through 1 gigabit interface on each machine (xen16-xen04)

Bonnie

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ppwn03.lcg.c 11000M   753  98 43095  11 28484  11  2881  97 130806  25  1053  88
Latency             13000us    2539ms    1990ms    8000us     295ms     299ms
Version  1.96       ------Sequential Create------ --------Random Create--------
ppwn03.lcg.cscs.ch  -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   333   1 +++++ +++   318   0   346   1 +++++ +++   321   0
Latency               273ms    1000us     213ms   28000us    1000us     408ms

-- PabloFernandez - 2011-09-05