<!-- keep this as a security measure: * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup #uncomment this if you want the page only be viewable by the internal people #* Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup --> ---+ Lustre Oss hardware ---++Using 3 Virident cards, 2 as metadataOnly, 1 as dataAndMetadata - 256K blocksize <verbatim> Sep 14 11:19 [root@wn120:tmpgpfs]# dd if=/dev/zero of=test bs=4096k count=10000 10000+0 records in 10000+0 records out 41943040000 bytes (42 GB) copied, 37.7524 seconds, 1.1 GB/s Sep 14 11:20 [root@wn120:tmpgpfs]# time tar xf linux-2.6.36.1.tar real 0m13.559s user 0m0.267s sys 0m3.727s make real 8m12.997s user 6m56.126s sys 0m58.973s </verbatim> ---++Using 3 virident cards as metadata + 3 raids, 2 on oss11, 1 on oss12 - 256K blocksize <verbatim> Sep 14 12:01 [root@wn120:tmpgpfs]# dd if=/dev/zero of=test bs=4096k count=10000 10000+0 records in 10000+0 records out 41943040000 bytes (42 GB) copied, 232.677 seconds, 180 MB/s Sep 14 12:27 [root@wn120:tmpgpfs]# time tar xf linux-2.6.36.1.tar real 0m37.709s user 0m0.051s sys 0m0.773s time make real 8m26.329s user 6m56.563s sys 0m58.123s dd test on three hosts Sep 14 12:42 [root@xen11:~]# dsh -w wn[120-122] /tmpgpfs/testgpfs.sh wn120: 10000+0 records in wn120: 10000+0 records out wn120: 41943040000 bytes (42 GB) copied, 666.36 seconds, 62.9 MB/s wn120: wn120: real 11m6.362s wn120: user 0m0.025s wn120: sys 0m20.165s wn121: 10000+0 records in wn121: 10000+0 records out wn121: 41943040000 bytes (42 GB) copied, 666.798 seconds, 62.9 MB/s wn121: wn121: real 11m6.800s wn121: user 0m0.011s wn121: sys 0m18.103s wn122: 10000+0 records in wn122: 10000+0 records out wn122: 41943040000 bytes (42 GB) copied, 671.016 seconds, 62.5 MB/s wn122: wn122: real 11m11.017s wn122: user 0m0.011s wn122: sys 0m17.614s </verbatim> on ten hosts, it's about 10 MB/s ---+ Xen16 machine ---++ dd <verbatim> [root@xen16 ~]# dd if=/dev/zero of=/virident/thing bs=8k count=1024k 1048576+0 records in 1048576+0 records out 8589934592 bytes (8.6 GB) copied, 13.6994 s, 627 MB/s [...] do some stuff [root@xen16 ~]# dd if=/virident/thing of=/dev/null ^[[6~16777216+0 records in 16777216+0 records out 8589934592 bytes (8.6 GB) copied, 18.8337 s, 456 MB/s </verbatim> ---++ Bonnie <verbatim> [root@xen16 ~]# bonnie++ -u root -s 60000 -r 0 -b -d /virident/ Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP xen16.lcg.cs 60000M 733 99 530557 94 298348 78 3522 99 722689 88 1078 112 Latency 11048us 14500us 194ms 2686us 4304us 5686us Version 1.96 ------Sequential Create------ --------Random Create-------- xen16.lcg.cscs.ch -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1599 6 +++++ +++ 2253 7 1777 8 +++++ +++ 2248 6 Latency 557ms 375us 4239us 6312us 17us 5872us </verbatim> ---++ Kernel compile <verbatim> [root@xen16 virident]# time tar -xf linux-2.6.36.1.tar real 0m1.828s user 0m0.123s sys 0m1.706s [root@xen16 linux-2.6.36.1]# time make real 7m53.340s user 6m46.717s sys 1m18.671s </verbatim> ---+ KVM over local Virident card (xen16) ---++ dd <verbatim> [root@xen16 ~]# dd if=/dev/zero of=/root/thing bs=4k count=1024k 1048576+0 records in 1048576+0 records out 4294967296 bytes (4.3 GB) copied, 25.7644 seconds, 167 MB/s [...] do some stuff [root@xen16 ~]# dd if=/root/thing of=/dev/null 8388608+0 records in 8388608+0 records out 4294967296 bytes (4.3 GB) copied, 12.177 seconds, 353 MB/s </verbatim> ---++ Bonnie <verbatim> [root@xen16 ~]# bonnie++ -u root -s 6000 -r 0 -b -d /root/ Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ppwn04.lcg.cs 6000M 870 99 139143 47 93778 37 3428 97 395061 38 3353 238 Latency 10000us 1418ms 828ms 3999us 10999us 73999us Version 1.96 ------Sequential Create------ --------Random Create-------- ppwn04.lcg.cscs.ch -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 905 3 +++++ +++ 890 2 888 4 +++++ +++ 892 2 Latency 30999us 1000us 21000us 32999us 1000us 16000us </verbatim> ---++ Kernel compile <verbatim> [root@xen16 virident]# time tar -xf linux-2.6.36.1.tar real 0m3.740s user 0m0.174s sys 0m2.973s [root@xen16 linux-2.6.36.1]# time make </verbatim> ---+ KVM over remote Virident card mounted on NFS through 1 gigabit interface on each machine (xen16-xen04) ---++ Bonnie <verbatim> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ppwn03.lcg.c 11000M 753 98 43095 11 28484 11 2881 97 130806 25 1053 88 Latency 13000us 2539ms 1990ms 8000us 295ms 299ms Version 1.96 ------Sequential Create------ --------Random Create-------- ppwn03.lcg.cscs.ch -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 333 1 +++++ +++ 318 0 346 1 +++++ +++ 321 0 Latency 273ms 1000us 213ms 28000us 1000us 408ms </verbatim> -- Main.PabloFernandez - 2011-09-05
This topic: LCGTier2
>
WebHome
>
TestsBoard
>
TestViridentCard
Topic revision: r6 - 2011-09-14 - JasonTemple
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback