Go to previous page / next page of Tier3 site log

14. 03. 2012 Network tests of the new 10GbE attached file servers for the IS5500

Tests between t3fs13 and t3fs14 to test the pure 10GbE connectivity

t3fs13 to t3fs14 directly connected by a 10GbE cable

An iperf test between the two machines shows varying throughputs from 3 to 7 Gbit/s. But when the iperf process is bound to a core using the taskset command the throughput is in the 7-9.5 Gb/s range. After the initial test, the iperf server process seems to go into a corrupted state. Any following connection usually is only in the 1 Gb/s range and the process cannot be killed by a simple kill (needs a SIGKILL). Therefore we ignore these followup tries and concentrate on the initial test conncetion.

The findings clearly indicate that the multicore architecture and the NIC scheduling can interfere badly with each other. There are some literature sources describing this behavior (http://arxiv.org/abs/1106.0443, http://kernel.org/doc/ols/2009/#169-184)

RTT was 0.19 ms

[root@t3fs13 ~]# iperf -s -w 512k -D
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 1.00 MByteTCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
Running Iperf Server as a daemon
The Iperf daemon process ID : 21802

[root@t3fs13 ~]# taskset -pc 01  21802
pid 21802's current affinity list: 0-23
pid 21802's new affinity list: 1

------------------------------------------------------------
Client connecting to 192.33.123.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.33.123.54 port 58796 connected with 192.33.123.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  11.0 GBytes  9.47 Gbits/sec
[root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10
------------------------------------------------------------
Client connecting to 192.33.123.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.33.123.54 port 58797 connected with 192.33.123.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   361 MBytes   303 Mbits/sec
[root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10
------------------------------------------------------------
Client connecting to 192.33.123.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.33.123.54 port 58798 connected with 192.33.123.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec   735 MBytes   611 Mbits/sec

t3fs13 to t3fs14 connected through two separate Switches over the backbone.

rtt min/avg/max/mdev = 0.132/0.176/0.226/0.031 ms

[root@t3fs13 ~]# iperf -s -w 512k -D
Running Iperf Server as a daemon
The Iperf daemon process ID : 21989
[root@t3fs13 ~]# taskset -pc 01 21989
pid 21989's current affinity list: 0-23
pid 21989's new affinity list: 1


[root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10
------------------------------------------------------------
Client connecting to 192.33.123.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.33.123.54 port 58806 connected with 192.33.123.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  10.4 GBytes  8.78 Gbits/sec


[root@t3fs13 ~]# iperf -s -w 512k -D
Running Iperf Server as a daemon
The Iperf daemon process ID : 22010
[root@t3fs13 ~]# taskset -pc 01 22010
pid 22010's current affinity list: 0-23
pid 22010's new affinity list: 1

[root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10
------------------------------------------------------------
Client connecting to 192.33.123.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.33.123.54 port 58810 connected with 192.33.123.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  9.97 GBytes  8.56 Gbits/sec
[root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10
------------------------------------------------------------
Client connecting to 192.33.123.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.33.123.54 port 58811 connected with 192.33.123.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.2 sec   404 MBytes   333 Mbits/sec
[root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10
------------------------------------------------------------
Client connecting to 192.33.123.53, TCP port 5001
TCP window size: 1.00 MByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 192.33.123.54 port 58812 connected with 192.33.123.53 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.4 sec   359 MBytes   290 Mbits/sec

Tests between t3fs13, t3fs14 and 10+10 1Gbit/s clients

I was testing if the 10Gbit/s bandwith is reached with 10 parallel 1Gbit/s connections from our WNs and I'm suffering this strange behaviour:

FS13
[  4]  0.0-10.0 sec   718 MBytes   602 Mbits/sec
[  5]  0.0-10.0 sec   880 MBytes   738 Mbits/sec
[  7]  0.0-10.0 sec   858 MBytes   719 Mbits/sec
[  8]  0.0-10.0 sec   787 MBytes   660 Mbits/sec
[ 10]  0.0-10.0 sec   881 MBytes   738 Mbits/sec
[ 11]  0.0-10.0 sec   787 MBytes   660 Mbits/sec
[ 13]  0.0-10.2 sec   897 MBytes   740 Mbits/sec
[  6]  0.0-10.0 sec   769 MBytes   643 Mbits/sec
[  9]  0.0-10.0 sec   813 MBytes   681 Mbits/sec
[ 12]  0.0-10.0 sec   832 MBytes   697 Mbits/sec
FS14
[  4]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[  5]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[  6]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[  7]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[  8]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[  9]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[ 10]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[ 11]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[ 12]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
[ 13]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec

To reproduce the error open 3 SSH connections:

• On t3fs13: start iperf -s -w 512k
• On t3fs14: start iperf -s -w 512k
• On t3admin01: start cexec wn:0-9 iperf -t 10 -c t3fs13 -w 256k
• On t3admin01: start cexec wn:0-9 iperf -t 10 -c t3fs14 -w 256k

When I've swapped the 2 Fibres between t3fs13 and t3fs14 the problem has moved as well. -- DerekFeichtinger - 2012-03-14

---

Go to previous page / next page of Tier3 site log