Throughput Tests with FNAL

The first few tests were done running iperf manually on Dec 1st 2006. On Dec 6 I tried to redo a full run with the help of my new script in an automated fashion, but the iperf service at FNAL ceased to work.

An automated run using the new scripts was done on Jan 11th 2007 using another host.

PS: There was a discussion on CMS hypernews (March 2007, "network tuning for SL4") that may point to a problem. It involves some intermediate routers that are not correctly able to handle the wscale TCP directive (look also at this article).

Dec 1st 2006, charley.fnal.gov

Route, RTT and packet loss

traceroute to charley.fnal.gov (131.225.2.3), 30 hops max, 38 byte packets
 1  148.187.33.2 (148.187.33.2)  0.429 ms  0.441 ms  0.375 ms
 2  148.187.32.4 (148.187.32.4)  0.205 ms  0.228 ms  0.234 ms
 3  swima2.cscs.ch (148.187.20.2)  0.491 ms  0.595 ms  0.497 ms
 4  swiEL2-10GE-1-4.switch.ch (130.59.37.77)  3.453 ms  3.300 ms  3.202 ms
 5  swiCE3-10GE-1-3.switch.ch (130.59.37.65)  4.094 ms  4.216 ms  3.851 ms
 6  swiCE2-10GE-1-4.switch.ch (130.59.36.209)  4.092 ms  4.087 ms  4.110 ms
 7  switch.rt1.gen.ch.geant2.net (62.40.124.21)  4.093 ms  4.153 ms  4.112 ms
 8  so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22)  12.218 ms  12.291 ms 12.231 ms
 9  esnet-wash-gw.rt1.fra.de.geant2.net (62.40.125.78)  118.081 ms  117.939 ms 117.997 ms
10  aoacr1-oc192-dccr1.es.net (134.55.219.46)  112.567 ms  112.513 ms  112.570 ms
11  chicr1-oc192-aoacr1.es.net (134.55.209.57)  132.696 ms  132.527 ms 132.570 ms
12  chislsdn1-chicr1.es.net (134.55.207.34)  132.683 ms  132.645 ms  132.571 ms
13  198.49.208.230 (198.49.208.230)  133.721 ms  133.812 ms  133.727 ms
14  vlan360.r-s-hub-fcc.fnal.gov (131.225.15.78)  133.982 ms  133.796 ms 133.852 ms
15  s-s-fapl.fnal.gov (131.225.15.29)  133.718 ms  137.085 ms  133.839 ms
16  charley.fnal.gov (131.225.2.3)  133.842 ms  133.921 ms  133.583 ms

The round trip time from an extended ping run was 133ms with 0 packet loss.

Influence of TCP window size and number of parallel streams

The upload graph (TO) looks ok and large TCP window sizes show huge improvements as is expected based on the large RTT of 133ms. The download graph shows up a major problem, because the rates are much too low. The best TCP window size of the series is 256KB and for all window sizes there is a linear increase of throughput with the number of streams. This points to a deeper problem (see below).

TCP based Problem for Downloads from FNAL (UNRESOLVED!)

This paragraph is based on a packet capture of an iperf run done with wireshark. The iperf command line used was:

 ./iperf -c charley.fnal.gov -p [hidden] -w 1024k -r -L [hidden] -m

The CSCS->FNAL test shows correct behavior. The initial window size of 17920 at the fnal end rises to 1501696- and it seems that a big part of the window is also used.

For the FNAL->CSCS test the TCP window grows from 8736 to 788384, which does not look too bad. but It seems that the window size is never really used:

The typical behavior is:
I receive from FNAL 4 ACK packets of 1460 length in short intervals (~3ms), where the last packet also has the PSH flag set. Then, after ~75 ms, the CSCS side sends out an ACK. It seems that the other side is not sending anything until this ACK has been received. The next packet from fnal arrives after ~133ms (the normal round trip time).

So, from the CSCS side it seems that the other side is never really using the available window (only 4*1460 bytes). Somehow, after sending a few packets, the other side is always waiting for an ACK from CSCS before continuing.

My FNAL->CSCS capture also shows many more PSH,ACK packets than the CSCS->FNAL one.

We need to get analysed captures done on the FNAL end to compare. It is not clear to me what is responsible for this kind of behavior, but it could well be that it's some network component between us.

Jan 11th 2007, fapl001.fnal.gov

Route, RTT and packet loss

traceroute to fapl001.fnal.gov (131.225.2.65), 30 hops max, 38 byte packets
 1  148.187.33.2 (148.187.33.2)  0.428 ms  0.449 ms  0.361 ms
 2  148.187.32.4 (148.187.32.4)  0.116 ms  0.102 ms  0.116 ms
 3  swima2.cscs.ch (148.187.20.2)  0.613 ms  0.473 ms  0.492 ms
 4  swiEL2-10GE-1-4.switch.ch (130.59.37.77)  3.114 ms  3.224 ms  3.240 ms
 5  swiCE3-10GE-1-3.switch.ch (130.59.37.65)  3.865 ms  4.102 ms  4.115 ms
 6  swiCE2-10GE-1-4.switch.ch (130.59.36.209)  4.114 ms  4.101 ms  3.989 ms
 7  switch.rt1.gen.ch.geant2.net (62.40.124.21)  4.113 ms  3.975 ms  4.115 ms
 8  so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22)  12.359 ms  12.095 ms  12.106 ms
 9  esnet-wash-gw.rt1.fra.de.geant2.net (62.40.125.78)  104.172 ms  104.159 ms  104.171 ms
10  aoacr1-oc192-dccr1.es.net (134.55.219.46)  107.428 ms  107.523 ms  107.547 ms
11  chicr1-oc192-aoacr1.es.net (134.55.209.57)  127.658 ms  127.627 ms  127.538 ms
12  chislsdn1-chicr1.es.net (134.55.207.34)  127.501 ms  127.925 ms  127.978 ms
13  198.49.208.230 (198.49.208.230)  128.968 ms  128.713 ms  128.777 ms
14  vlan360.r-s-hub-fcc.fnal.gov (131.225.15.78)  128.652 ms  128.507 ms  128.534 ms
15  s-s-fapl.fnal.gov (131.225.15.29)  128.777 ms  135.675 ms  128.507 ms
16  fapl001.fnal.gov (131.225.2.65)  128.621 ms  128.725 ms  128.684 ms

The round trip time from an extended ping run was 128ms with 0 packet loss:

20 packets transmitted, 20 received, 0% packet loss, time 19197ms
rtt min/avg/max/mdev = 128.420/128.643/129.353/0.225 ms, pipe 2

Influence of TCP window size and number of parallel streams

The tests with fapl001.fnal.gov give a much more consistent picture. Download and upload rates are of a comparable magnitude. The huge variations seen in these measurements are probably due to other traffic.

May 1st 2007, fapl001.fnal.gov

packet loss for route elements

An mtr run showed

HOST                                    LOSS  RCVD SENT    BEST     AVG   WORST
148.187.33.2                              0%    30   30    0.36    0.46    0.80
148.187.32.4                              0%    30   30    0.10    0.20    0.46
swima2.cscs.ch                           10%    27   30    0.38    0.48    0.67
swiEL2-10GE-1-4.switch.ch                 4%    29   30    3.12    5.38   63.96
swiCE3-10GE-1-3.switch.ch                 0%    30   30    3.84    3.96    4.13
swiCE2-10GE-1-4.switch.ch                 0%    30   30    3.97    5.97   37.20
switch.rt1.gen.ch.geant2.net              0%    30   30    4.00    4.10    4.42
so-7-2-0.rt1.fra.de.geant2.net            0%    30   30   12.10   12.22   12.56
esnet-wash-gw.rt1.fra.de.geant2.net       0%    30   30  104.30  105.17  114.19
washsdn1-sdn2-washcr1.es.net              0%    30   30  104.42  115.45  271.71
chislmr1-washsdn1.es.net                  0%    30   30  122.05  128.71  250.88
chislsdn1-chislmr1.es.net                 0%    30   30  122.02  122.87  132.50
fnalmr1-chislsdn1.es.net                  0%    30   30  123.15  123.31  123.51
198.49.208.230                            0%    30   30  123.18  123.44  126.64
vlan360.r-s-hub-fcc.fnal.gov              0%    30   30  123.15  123.26  123.46
s-s-fapl.fnal.gov                         4%    29   30  123.21  123.40  123.97
fapl001.fnal.gov                          4%    29   30  123.10  123.27  123.59

-- DerekFeichtinger - 01 May 2007