Testing of the X4500's interface bonding
Local port to switch port mappings
These mappings are needed for correctly configuring the switch
For incoming connections, the 4 bonded interfaces of each X4500 must be known to the Switch, so that these connections are balanced over the interfaces.
The tests used my
IperfLoadTest test suite.
Test setup:
- 3 X4500 servers (Thumpers) with 4*1Gb/s aggregated (bonded) interfaces (File Servers). t3fs01 and t3fs03 were running Solaris 10 while t3fs02 was running OpenSolaris
- 7 X4150 servers with 1*1Gb/s interface (Worker Nodes)
Tests for worker nodes sending data to fileservers
This is testing the switch and the hash function it uses to balance connections between the target interfaces on every thumper. The CISCO switches we use are regrettably only able to do IP or MAC based hashing.
Measurements for worker nodes sending
Every measurement was run for 60s with client processes on the worker nodes sending TCP streams to receiving processes on the fileservers.
7 processes on 7 worker nodes sending to 1 Fileserver:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.87:32866 192.33.123.41:8001 3990.00 571.00
192.33.123.81:32881 192.33.123.41:8001 5950.00 851.00
192.33.123.84:32865 192.33.123.41:8001 6000.00 859.00
192.33.123.86:32865 192.33.123.41:8001 1410.00 201.00
192.33.123.83:32861 192.33.123.41:8001 2590.00 371.00
192.33.123.82:32862 192.33.123.41:8001 2590.00 370.00
192.33.123.85:32868 192.33.123.41:8001 2590.00 370.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.41:8001 25120.00 3593.00 7
Total:
--------------------------------------------
Rate: 3593 Mbits/sec
Data sent: 25120 MBytes
7 processes on 7 worker nodes sending to 3 Fileservers:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.83:32831 192.33.123.43:8001 3650.00 523.00
192.33.123.81:32851 192.33.123.43:8001 3630.00 520.00
192.33.123.84:32835 192.33.123.41:8001 5380.00 770.00
192.33.123.87:32836 192.33.123.42:8001 5430.00 777.00
192.33.123.85:32838 192.33.123.42:8001 5350.00 766.00
192.33.123.82:32832 192.33.123.41:8001 3450.00 494.00
192.33.123.86:32835 192.33.123.41:8001 3010.00 431.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.43:8001 7280.00 1043.00 2
192.33.123.41:8001 11840.00 1695.00 3
192.33.123.42:8001 10780.00 1543.00 2
Total:
--------------------------------------------
Rate: 4281 Mbits/sec
Data sent: 29900 MBytes
14 processes on 7 worker nodes sending to 3 Fileservers:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.82:32834 192.33.123.41:8001 2150.00 308.00
192.33.123.83:32831 192.33.123.43:8001 3360.00 480.00
192.33.123.87:32836 192.33.123.42:8001 2690.00 385.00
192.33.123.81:32851 192.33.123.43:8001 3390.00 485.00
192.33.123.86:32835 192.33.123.41:8001 1950.00 279.00
192.33.123.87:32838 192.33.123.41:8001 3770.00 540.00
192.33.123.86:32837 192.33.123.42:8001 3580.00 513.00
192.33.123.83:32833 192.33.123.42:8001 2890.00 414.00
192.33.123.85:32840 192.33.123.42:8001 2830.00 406.00
192.33.123.81:32853 192.33.123.43:8001 2870.00 411.00
192.33.123.84:32835 192.33.123.41:8001 3250.00 465.00
192.33.123.84:32837 192.33.123.43:8001 3250.00 465.00
192.33.123.85:32838 192.33.123.43:8001 3700.00 529.00
192.33.123.82:32832 192.33.123.41:8001 2470.00 354.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.43:8001 16570.00 2370.00 5
192.33.123.41:8001 13590.00 1946.00 5
192.33.123.42:8001 11990.00 1718.00 4
Total:
--------------------------------------------
Rate: 6034 Mbits/sec
Data sent: 42150 MBytes
21 processes on 7 worker nodes sending to 3 Fileservers:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.82:32837 192.33.123.41:8001 1820.00 260.00
192.33.123.84:32840 192.33.123.43:8001 1780.00 255.00
192.33.123.87:32836 192.33.123.41:8001 1960.00 281.00
192.33.123.81:32851 192.33.123.43:8001 1960.00 280.00
192.33.123.87:32838 192.33.123.42:8001 2250.00 322.00
192.33.123.87:32841 192.33.123.41:8001 2360.00 338.00
192.33.123.83:32833 192.33.123.41:8001 2250.00 322.00
192.33.123.85:32840 192.33.123.42:8001 1530.00 219.00
192.33.123.84:32837 192.33.123.43:8001 3010.00 431.00
192.33.123.85:32838 192.33.123.43:8001 3380.00 483.00
192.33.123.83:32831 192.33.123.42:8001 1790.00 256.00
192.33.123.82:32834 192.33.123.41:8001 2480.00 355.00
192.33.123.85:32843 192.33.123.42:8001 1530.00 219.00
192.33.123.81:32856 192.33.123.43:8001 1960.00 280.00
192.33.123.86:32840 192.33.123.41:8001 2280.00 327.00
192.33.123.86:32835 192.33.123.42:8001 1760.00 252.00
192.33.123.86:32837 192.33.123.42:8001 1760.00 252.00
192.33.123.81:32853 192.33.123.43:8001 2660.00 381.00
192.33.123.84:32835 192.33.123.41:8001 1780.00 255.00
192.33.123.83:32836 192.33.123.42:8001 2540.00 363.00
192.33.123.82:32832 192.33.123.43:8001 2230.00 319.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.43:8001 16980.00 2429.00 7
192.33.123.41:8001 14930.00 2138.00 7
192.33.123.42:8001 13160.00 1883.00 7
Total:
--------------------------------------------
Rate: 6450 Mbits/sec
Data sent: 45070 MBytes
28 processes on 7 worker nodes sending to 3 Fileservers:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.82:32837 192.33.123.43:8001 1580.00 226.00
192.33.123.84:32840 192.33.123.41:8001 1640.00 235.00
192.33.123.87:32845 192.33.123.41:8001 2220.00 318.00
192.33.123.87:32836 192.33.123.43:8001 1440.00 206.00
192.33.123.81:32851 192.33.123.41:8001 1650.00 236.00
192.33.123.85:32847 192.33.123.43:8001 1330.00 191.00
192.33.123.87:32838 192.33.123.43:8001 1440.00 206.00
192.33.123.87:32841 192.33.123.42:8001 1470.00 211.00
192.33.123.83:32833 192.33.123.42:8001 1470.00 210.00
192.33.123.85:32840 192.33.123.43:8001 1330.00 191.00
192.33.123.84:32837 192.33.123.41:8001 1640.00 235.00
192.33.123.81:32860 192.33.123.41:8001 1650.00 236.00
192.33.123.85:32838 192.33.123.42:8001 1430.00 205.00
192.33.123.82:32841 192.33.123.41:8001 2880.00 413.00
192.33.123.84:32844 192.33.123.42:8001 1640.00 235.00
192.33.123.83:32831 192.33.123.41:8001 1470.00 210.00
192.33.123.82:32834 192.33.123.43:8001 1060.00 151.00
192.33.123.85:32843 192.33.123.42:8001 1430.00 205.00
192.33.123.81:32856 192.33.123.41:8001 1650.00 236.00
192.33.123.86:32840 192.33.123.42:8001 1330.00 190.00
192.33.123.86:32835 192.33.123.43:8001 1270.00 182.00
192.33.123.86:32837 192.33.123.42:8001 1330.00 190.00
192.33.123.86:32844 192.33.123.41:8001 2600.00 372.00
192.33.123.81:32853 192.33.123.43:8001 1640.00 235.00
192.33.123.84:32835 192.33.123.42:8001 1640.00 235.00
192.33.123.83:32836 192.33.123.43:8001 1430.00 205.00
192.33.123.82:32832 192.33.123.42:8001 1050.00 151.00
192.33.123.83:32840 192.33.123.41:8001 2200.00 316.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.43:8001 12520.00 1793.00 9
192.33.123.41:8001 19600.00 2807.00 10
192.33.123.42:8001 12790.00 1832.00 9
Total:
--------------------------------------------
Rate: 6432 Mbits/sec
Data sent: 44910 MBytes
Tests for worker nodes reading data from fileservers
Measurements for worker nodes reading
4 processes on 1 fileserver sending to 4 worker nodes:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.41:33536 192.33.123.85:8001 3300.00 473.00
192.33.123.41:33539 192.33.123.83:8001 3220.00 461.00
192.33.123.41:33537 192.33.123.82:8001 6390.00 915.00
192.33.123.41:33538 192.33.123.81:8001 6300.00 902.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.85:8001 3300.00 473.00 1
192.33.123.81:8001 6300.00 902.00 1
192.33.123.83:8001 3220.00 461.00 1
192.33.123.82:8001 6390.00 915.00 1
Total:
--------------------------------------------
Rate: 2751 Mbits/sec
Data sent: 19210 MBytes
The less than expected throughput of 2.751 Mbits/s may be related to the non negligible processor load for the 4 iperf processes on the file server:
Show Hide
last pid: 10517; load avg: 1.68, 0.68, 0.39; up 13+18:24:39 11:49:32
63 processes: 58 sleeping, 1 running, 4 on cpu
CPU states: 0.0% idle, 42.8% user, 57.2% kernel, 0.0% iowait, 0.0% swap
Memory: 16G phys mem, 8965M free mem, 2000M swap, 2000M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
10517 root 3 10 0 3840K 2284K cpu 0:33 108% iperf_sun
10516 root 3 31 0 3840K 2284K cpu 0:31 104% iperf_sun
10514 root 3 20 0 3840K 2284K cpu 0:31 102% iperf_sun
10515 root 3 10 0 3840K 2284K run 0:26 78.66% iperf_sun
2 processes on 1 fileserver sending to 2 worker nodes:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.41:33553 192.33.123.86:8001 6600.00 945.00
192.33.123.41:33554 192.33.123.82:8001 6600.00 945.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.82:8001 6600.00 945.00 1
192.33.123.86:8001 6600.00 945.00 1
Total:
--------------------------------------------
Rate: 1890 Mbits/sec
Data sent: 13200 MBytes
3 processes on 1 fileserver sending to 3 worker nodes:
Show Hide
Config: mode=tcp time=60
Clients:
--------------------------------------------
Client Server Total Rate
MBytes Mbits/s
192.33.123.41:33553 192.33.123.86:8001 6460.00 925.00
192.33.123.41:33554 192.33.123.82:8001 3250.00 466.00
192.33.123.41:33557 192.33.123.85:8001 3250.00 466.00
Servers:
--------------------------------------------
Server Total Rate clients
MBytes Mbits/s
192.33.123.85:8001 3250.00 466.00 1
192.33.123.82:8001 3250.00 466.00 1
192.33.123.86:8001 6460.00 925.00 1
Total:
--------------------------------------------
Rate: 1857 Mbits/sec
Data sent: 12960 MBytes
Processor load:
Show Hide
last pid: 10557; load avg: 1.02, 0.74, 0.57; up 13+18:38:34 12:03:27
61 processes: 57 sleeping, 4 on cpu
CPU states: 15.7% idle, 43.0% user, 41.3% kernel, 0.0% iowait, 0.0% swap
Memory: 16G phys mem, 8816M free mem, 2000M swap, 2000M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
10555 root 3 0 0 3840K 2284K cpu 0:26 115% iperf_sun
10557 root 3 10 0 3840K 2284K cpu 0:24 107% iperf_sun
10556 root 3 0 0 3840K 2284K cpu 0:24 107% iperf_sun
--
DerekFeichtinger - 08 Sep 2008