Tags:
tag this topic
create new tag
view all tags
<!-- keep this as a security measure: #uncomment if the subject should only be modifiable by the listed groups * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.CMSAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.CMSAdminGroup #uncomment this if you want the page only be viewable by the listed groups # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.CMSAdminGroup --> %TOC% %ICON{arrowleft}% Go to [[CMSTier3LogXX][previous page]] / [[CMSTier3LogXX][next page]] of Tier3 site log %M% ---+!! 14. 03. 2012 Network tests of the new 10GbE attached file servers for the IS5500 ---++ Tests between t3fs13 and t3fs14 to test the pure 10GbE connectivity ---+++ t3fs13 to t3fs14 directly connected by a 10GbE cable An iperf test between the two machines shows varying throughputs from 3 to 7 Gbit/s. But when the iperf process is bound to a core using the taskset command the throughput is in the 7-9.5 Gb/s range. After the initial test, the iperf server process seems to go into a corrupted state. Any following connection usually is only in the 1 Gb/s range and the process cannot be killed by a simple kill (needs a SIGKILL). Therefore we ignore these followup tries and concentrate on the initial test conncetion. The findings clearly indicate that the multicore architecture and the NIC scheduling can interfere badly with each other. There are some literature sources describing this behavior (http://arxiv.org/abs/1106.0443, http://kernel.org/doc/ols/2009/#169-184) RTT was 0.19 ms <pre> [root@t3fs13 ~]# iperf -s -w 512k -D ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 1.00 MByteTCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ Running Iperf Server as a daemon The Iperf daemon process ID : 21802 [root@t3fs13 ~]# taskset -pc 01 21802 pid 21802's current affinity list: 0-23 pid 21802's new affinity list: 1 </pre> <pre> ------------------------------------------------------------ Client connecting to 192.33.123.53, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 3] local 192.33.123.54 port 58796 connected with 192.33.123.53 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 11.0 GBytes 9.47 Gbits/sec [root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10 ------------------------------------------------------------ Client connecting to 192.33.123.53, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 3] local 192.33.123.54 port 58797 connected with 192.33.123.53 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 361 MBytes 303 Mbits/sec [root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10 ------------------------------------------------------------ Client connecting to 192.33.123.53, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 3] local 192.33.123.54 port 58798 connected with 192.33.123.53 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 735 MBytes 611 Mbits/sec </pre> ---+++ t3fs13 to t3fs14 connected through two separate Switches over the backbone. rtt min/avg/max/mdev = 0.132/0.176/0.226/0.031 ms <pre> [root@t3fs13 ~]# iperf -s -w 512k -D Running Iperf Server as a daemon The Iperf daemon process ID : 21989 [root@t3fs13 ~]# taskset -pc 01 21989 pid 21989's current affinity list: 0-23 pid 21989's new affinity list: 1 [root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10 ------------------------------------------------------------ Client connecting to 192.33.123.53, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 3] local 192.33.123.54 port 58806 connected with 192.33.123.53 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 10.4 GBytes 8.78 Gbits/sec [root@t3fs13 ~]# iperf -s -w 512k -D Running Iperf Server as a daemon The Iperf daemon process ID : 22010 [root@t3fs13 ~]# taskset -pc 01 22010 pid 22010's current affinity list: 0-23 pid 22010's new affinity list: 1 [root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10 ------------------------------------------------------------ Client connecting to 192.33.123.53, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 3] local 192.33.123.54 port 58810 connected with 192.33.123.53 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 9.97 GBytes 8.56 Gbits/sec [root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10 ------------------------------------------------------------ Client connecting to 192.33.123.53, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 3] local 192.33.123.54 port 58811 connected with 192.33.123.53 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.2 sec 404 MBytes 333 Mbits/sec [root@t3fs14 ~]# iperf -c 192.33.123.53 -w 512k -t 10 ------------------------------------------------------------ Client connecting to 192.33.123.53, TCP port 5001 TCP window size: 1.00 MByte (WARNING: requested 512 KByte) ------------------------------------------------------------ [ 3] local 192.33.123.54 port 58812 connected with 192.33.123.53 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.4 sec 359 MBytes 290 Mbits/sec </pre> ---++ Tests between t3fs13, t3fs14 and 10+10 1Gbit/s clients I was testing if the 10Gbit/s bandwith is reached with 10 parallel 1Gbit/s connections from our WNs and I'm suffering this strange behaviour: <pre> FS13 [ 4] 0.0-10.0 sec 718 MBytes 602 Mbits/sec [ 5] 0.0-10.0 sec 880 MBytes 738 Mbits/sec [ 7] 0.0-10.0 sec 858 MBytes 719 Mbits/sec [ 8] 0.0-10.0 sec 787 MBytes 660 Mbits/sec [ 10] 0.0-10.0 sec 881 MBytes 738 Mbits/sec [ 11] 0.0-10.0 sec 787 MBytes 660 Mbits/sec [ 13] 0.0-10.2 sec 897 MBytes 740 Mbits/sec [ 6] 0.0-10.0 sec 769 MBytes 643 Mbits/sec [ 9] 0.0-10.0 sec 813 MBytes 681 Mbits/sec [ 12] 0.0-10.0 sec 832 MBytes 697 Mbits/sec FS14 [ 4] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 5] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 6] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 7] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 8] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 9] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 10] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 11] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 12] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec [ 13] 0.0-10.0 sec 1.11 GBytes 949 Mbits/sec </pre> To reproduce the error open 3 SSH connections: * On =t3fs13=: start =iperf -s -w 512k= * On =t3fs14=: start =iperf -s -w 512k= * On =t3admin01=: start =cexec wn:0-9 iperf -t 10 -c t3fs13 -w 256k= * On =t3admin01=: start =cexec wn:0-9 iperf -t 10 -c t3fs14 -w 256k= When I've swapped the 2 Fibres between =t3fs13= and =t3fs14= the problem has moved as well. -- Main.DerekFeichtinger - 2012-03-14 ---------------- %ICON{arrowleft}% Go to [[CMSTier3LogXX][previous page]] / [[CMSTier3LogXX][next page]] of Tier3 site log %M%
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r2
<
r1
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r2 - 2012-03-15
-
FabioMartinelli
CmsTier3
Log In
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
User Pages
Main Page
Policies
Monitoring Storage Space
Monitoring Slurm Usage
Physics Groups
Steering Board Meetings
Admin Pages
AdminArea
Cluster Specs
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
CmsTier3 Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
E
dit
A
ttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback