Tags:
create new tag
view all tags

Thumper local port to switch port mappings and trunking tests

Local port to switch port mappings

These mappings are needed for correctly configuring the switch

Host Name local interface switch port
t3fs01 0 10
1 9
2 12
3 11
t3fs02 0 13
1 14
2 15
3 16
t3fs03 0 8
1 6
2 7
3 5
t3fs04 0 1
1 4
2 2
3 3
t3fs05 0 23
1 21
2 24
3 22
t3fs06 0 17
1 20
2 19
3 18

For incoming connections, the 4 bonded interfaces of each X4500 must be known to the Switch, so that these connections are balanced over the interfaces.

Testing of the X4500's interface bonding

The tests used my IperfLoadTest test suite.

Test setup:

  • 3 X4500 servers (Thumpers) with 4*1Gb/s aggregated (bonded) interfaces (File Servers). t3fs01 and t3fs03 were running Solaris 10 while t3fs02 was running OpenSolaris
  • 7 X4150 servers with 1*1Gb/s interface (Worker Nodes)

Tests for worker nodes sending data to fileservers

This is testing the switch and the hash function it uses to balance connections between the target interfaces on every thumper. The CISCO switches we use are regrettably only able to do IP or MAC based hashing.

throughput measurements summary table
Sending Processes Sending worker nodes Receiving servers Rate (MBit/s)
  (each with 1GBit/s) (each with 4*1GBit/s)  
7 7 1 3593
7 7 3 4281
14 7 3 6034
21 7 3 6450
28 7 3 6432

Measurements for worker nodes sending

Every measurement was run for 60s with client processes on the worker nodes sending TCP streams to receiving processes on the fileservers.

7 processes on 7 worker nodes sending to 1 Fileserver: Show Hide

Config: mode=tcp time=60


Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.87:32866  192.33.123.41:8001 3990.00  571.00
192.33.123.81:32881  192.33.123.41:8001 5950.00  851.00
192.33.123.84:32865  192.33.123.41:8001 6000.00  859.00
192.33.123.86:32865  192.33.123.41:8001 1410.00  201.00
192.33.123.83:32861  192.33.123.41:8001 2590.00  371.00
192.33.123.82:32862  192.33.123.41:8001 2590.00  370.00
192.33.123.85:32868  192.33.123.41:8001 2590.00  370.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.41:8001 25120.00 3593.00     7

Total:
--------------------------------------------
Rate: 3593 Mbits/sec
Data sent: 25120 MBytes

7 processes on 7 worker nodes sending to 3 Fileservers: Show Hide

Config: mode=tcp time=60

Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.83:32831  192.33.123.43:8001 3650.00  523.00
192.33.123.81:32851  192.33.123.43:8001 3630.00  520.00
192.33.123.84:32835  192.33.123.41:8001 5380.00  770.00
192.33.123.87:32836  192.33.123.42:8001 5430.00  777.00
192.33.123.85:32838  192.33.123.42:8001 5350.00  766.00
192.33.123.82:32832  192.33.123.41:8001 3450.00  494.00
192.33.123.86:32835  192.33.123.41:8001 3010.00  431.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.43:8001 7280.00 1043.00     2
192.33.123.41:8001 11840.00 1695.00     3
192.33.123.42:8001 10780.00 1543.00     2

Total:
--------------------------------------------
Rate: 4281 Mbits/sec
Data sent: 29900 MBytes

14 processes on 7 worker nodes sending to 3 Fileservers: Show Hide

Config: mode=tcp time=60


Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.82:32834  192.33.123.41:8001 2150.00  308.00
192.33.123.83:32831  192.33.123.43:8001 3360.00  480.00
192.33.123.87:32836  192.33.123.42:8001 2690.00  385.00
192.33.123.81:32851  192.33.123.43:8001 3390.00  485.00
192.33.123.86:32835  192.33.123.41:8001 1950.00  279.00
192.33.123.87:32838  192.33.123.41:8001 3770.00  540.00
192.33.123.86:32837  192.33.123.42:8001 3580.00  513.00
192.33.123.83:32833  192.33.123.42:8001 2890.00  414.00
192.33.123.85:32840  192.33.123.42:8001 2830.00  406.00
192.33.123.81:32853  192.33.123.43:8001 2870.00  411.00
192.33.123.84:32835  192.33.123.41:8001 3250.00  465.00
192.33.123.84:32837  192.33.123.43:8001 3250.00  465.00
192.33.123.85:32838  192.33.123.43:8001 3700.00  529.00
192.33.123.82:32832  192.33.123.41:8001 2470.00  354.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.43:8001 16570.00 2370.00     5
192.33.123.41:8001 13590.00 1946.00     5
192.33.123.42:8001 11990.00 1718.00     4

Total:
--------------------------------------------
Rate: 6034 Mbits/sec
Data sent: 42150 MBytes

21 processes on 7 worker nodes sending to 3 Fileservers: Show Hide

Config: mode=tcp time=60

Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.82:32837  192.33.123.41:8001 1820.00  260.00
192.33.123.84:32840  192.33.123.43:8001 1780.00  255.00
192.33.123.87:32836  192.33.123.41:8001 1960.00  281.00
192.33.123.81:32851  192.33.123.43:8001 1960.00  280.00
192.33.123.87:32838  192.33.123.42:8001 2250.00  322.00
192.33.123.87:32841  192.33.123.41:8001 2360.00  338.00
192.33.123.83:32833  192.33.123.41:8001 2250.00  322.00
192.33.123.85:32840  192.33.123.42:8001 1530.00  219.00
192.33.123.84:32837  192.33.123.43:8001 3010.00  431.00
192.33.123.85:32838  192.33.123.43:8001 3380.00  483.00
192.33.123.83:32831  192.33.123.42:8001 1790.00  256.00
192.33.123.82:32834  192.33.123.41:8001 2480.00  355.00
192.33.123.85:32843  192.33.123.42:8001 1530.00  219.00
192.33.123.81:32856  192.33.123.43:8001 1960.00  280.00
192.33.123.86:32840  192.33.123.41:8001 2280.00  327.00
192.33.123.86:32835  192.33.123.42:8001 1760.00  252.00
192.33.123.86:32837  192.33.123.42:8001 1760.00  252.00
192.33.123.81:32853  192.33.123.43:8001 2660.00  381.00
192.33.123.84:32835  192.33.123.41:8001 1780.00  255.00
192.33.123.83:32836  192.33.123.42:8001 2540.00  363.00
192.33.123.82:32832  192.33.123.43:8001 2230.00  319.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.43:8001 16980.00 2429.00     7
192.33.123.41:8001 14930.00 2138.00     7
192.33.123.42:8001 13160.00 1883.00     7

Total:
--------------------------------------------
Rate: 6450 Mbits/sec
Data sent: 45070 MBytes

28 processes on 7 worker nodes sending to 3 Fileservers: Show Hide

Config: mode=tcp time=60


Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.82:32837  192.33.123.43:8001 1580.00  226.00
192.33.123.84:32840  192.33.123.41:8001 1640.00  235.00
192.33.123.87:32845  192.33.123.41:8001 2220.00  318.00
192.33.123.87:32836  192.33.123.43:8001 1440.00  206.00
192.33.123.81:32851  192.33.123.41:8001 1650.00  236.00
192.33.123.85:32847  192.33.123.43:8001 1330.00  191.00
192.33.123.87:32838  192.33.123.43:8001 1440.00  206.00
192.33.123.87:32841  192.33.123.42:8001 1470.00  211.00
192.33.123.83:32833  192.33.123.42:8001 1470.00  210.00
192.33.123.85:32840  192.33.123.43:8001 1330.00  191.00
192.33.123.84:32837  192.33.123.41:8001 1640.00  235.00
192.33.123.81:32860  192.33.123.41:8001 1650.00  236.00
192.33.123.85:32838  192.33.123.42:8001 1430.00  205.00
192.33.123.82:32841  192.33.123.41:8001 2880.00  413.00
192.33.123.84:32844  192.33.123.42:8001 1640.00  235.00
192.33.123.83:32831  192.33.123.41:8001 1470.00  210.00
192.33.123.82:32834  192.33.123.43:8001 1060.00  151.00
192.33.123.85:32843  192.33.123.42:8001 1430.00  205.00
192.33.123.81:32856  192.33.123.41:8001 1650.00  236.00
192.33.123.86:32840  192.33.123.42:8001 1330.00  190.00
192.33.123.86:32835  192.33.123.43:8001 1270.00  182.00
192.33.123.86:32837  192.33.123.42:8001 1330.00  190.00
192.33.123.86:32844  192.33.123.41:8001 2600.00  372.00
192.33.123.81:32853  192.33.123.43:8001 1640.00  235.00
192.33.123.84:32835  192.33.123.42:8001 1640.00  235.00
192.33.123.83:32836  192.33.123.43:8001 1430.00  205.00
192.33.123.82:32832  192.33.123.42:8001 1050.00  151.00
192.33.123.83:32840  192.33.123.41:8001 2200.00  316.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.43:8001 12520.00 1793.00     9
192.33.123.41:8001 19600.00 2807.00    10
192.33.123.42:8001 12790.00 1832.00     9

Total:
--------------------------------------------
Rate: 6432 Mbits/sec
Data sent: 44910 MBytes

Tests for worker nodes reading data from fileservers

throughput measurements summary table
Sending Processes Sending servers Receiving worker nodes Rate (MBit/s)
  (each with 1GBit/s) (each with 4*1GBit/s)  
4 1 4 2751
3 1 3 1857
2 1 2 1890

Measurements for worker nodes reading

4 processes on 1 fileserver sending to 4 worker nodes: Show Hide

Config: mode=tcp time=60


Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.41:33536  192.33.123.85:8001 3300.00  473.00
192.33.123.41:33539  192.33.123.83:8001 3220.00  461.00
192.33.123.41:33537  192.33.123.82:8001 6390.00  915.00
192.33.123.41:33538  192.33.123.81:8001 6300.00  902.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.85:8001 3300.00  473.00     1
192.33.123.81:8001 6300.00  902.00     1
192.33.123.83:8001 3220.00  461.00     1
192.33.123.82:8001 6390.00  915.00     1

Total:
--------------------------------------------
Rate: 2751 Mbits/sec
Data sent: 19210 MBytes

The less than expected throughput of 2.751 Mbits/s may be related to the non negligible processor load for the 4 iperf processes on the file server: Show Hide

last pid: 10517;  load avg:  1.68,  0.68,  0.39;       up 13+18:24:39                                              11:49:32
63 processes: 58 sleeping, 1 running, 4 on cpu
CPU states:  0.0% idle, 42.8% user, 57.2% kernel,  0.0% iowait,  0.0% swap
Memory: 16G phys mem, 8965M free mem, 2000M swap, 2000M free swap

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 10517 root       3  10    0 3840K 2284K cpu      0:33   108% iperf_sun
 10516 root       3  31    0 3840K 2284K cpu      0:31   104% iperf_sun
 10514 root       3  20    0 3840K 2284K cpu      0:31   102% iperf_sun
 10515 root       3  10    0 3840K 2284K run      0:26 78.66% iperf_sun

2 processes on 1 fileserver sending to 2 worker nodes: Show Hide

Config: mode=tcp time=60

Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.41:33553  192.33.123.86:8001 6600.00  945.00
192.33.123.41:33554  192.33.123.82:8001 6600.00  945.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.82:8001 6600.00  945.00     1
192.33.123.86:8001 6600.00  945.00     1

Total:
--------------------------------------------
Rate: 1890 Mbits/sec
Data sent: 13200 MBytes

3 processes on 1 fileserver sending to 3 worker nodes: Show Hide

Config: mode=tcp time=60

Clients:
--------------------------------------------
     Client             Server       Total   Rate
                                     MBytes  Mbits/s
192.33.123.41:33553  192.33.123.86:8001 6460.00  925.00
192.33.123.41:33554  192.33.123.82:8001 3250.00  466.00
192.33.123.41:33557  192.33.123.85:8001 3250.00  466.00

Servers:
--------------------------------------------
     Server       Total   Rate     clients
                  MBytes  Mbits/s
192.33.123.85:8001 3250.00  466.00     1
192.33.123.82:8001 3250.00  466.00     1
192.33.123.86:8001 6460.00  925.00     1

Total:
--------------------------------------------
Rate: 1857 Mbits/sec
Data sent: 12960 MBytes

Processor load: Show Hide

last pid: 10557;  load avg:  1.02,  0.74,  0.57;       up 13+18:38:34                                              12:03:27
61 processes: 57 sleeping, 4 on cpu
CPU states: 15.7% idle, 43.0% user, 41.3% kernel,  0.0% iowait,  0.0% swap
Memory: 16G phys mem, 8816M free mem, 2000M swap, 2000M free swap

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 10555 root       3   0    0 3840K 2284K cpu      0:26   115% iperf_sun
 10557 root       3  10    0 3840K 2284K cpu      0:24   107% iperf_sun
 10556 root       3   0    0 3840K 2284K cpu      0:24   107% iperf_sun

-- DerekFeichtinger - 08 Sep 2008

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r8 - 2019-01-03 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback