Skip to content

Instantly share code, notes, and snippets.

@gangliao
Last active June 14, 2018 09:22
Show Gist options
  • Save gangliao/ed9fd49f46e270d3e40548c4d6d7653b to your computer and use it in GitHub Desktop.
Save gangliao/ed9fd49f46e270d3e40548c4d6d7653b to your computer and use it in GitHub Desktop.

Freeflow

TCP Physical Bandwidth

ethtool eth0 | grep Speed
# 	Speed: 1000Mb/s

demo test

bare metal

10.141.170.36:

# Terminal 1
yum install -y iperf3
iperf3 -s
# Terminal 2
iperf3 -c 10.141.170.36 -V
Linux nmyjs_170_36 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64
Control connection MSS 21888
Time: Thu, 14 Jun 2018 09:02:42 GMT
Connecting to host 10.141.170.36, port 5201
      Cookie: nmyjs_170_36.1528966962.822802.0eb91
      TCP MSS: 21888 (default)
[  4] local 10.141.170.36 port 59404 connected to 10.141.170.36 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.39 GBytes  20.5 Gbits/sec    0   1.06 MBytes
[  4]   1.00-2.00   sec  3.46 GBytes  29.7 Gbits/sec    0   1.06 MBytes
[  4]   2.00-3.00   sec  4.09 GBytes  35.2 Gbits/sec    0   1.12 MBytes
[  4]   3.00-4.00   sec  3.83 GBytes  32.9 Gbits/sec    0   1.12 MBytes
[  4]   4.00-5.00   sec  3.29 GBytes  28.3 Gbits/sec    0   1.25 MBytes
[  4]   5.00-6.00   sec  4.45 GBytes  38.3 Gbits/sec    0   1.25 MBytes
[  4]   6.00-7.00   sec  4.44 GBytes  38.1 Gbits/sec    0   1.25 MBytes
[  4]   7.00-8.00   sec  4.33 GBytes  37.2 Gbits/sec    0   1.87 MBytes
[  4]   8.00-9.00   sec  4.61 GBytes  39.6 Gbits/sec    0   1.87 MBytes
[  4]   9.00-10.00  sec  4.65 GBytes  40.0 Gbits/sec    0   1.87 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  39.6 GBytes  34.0 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  39.6 GBytes  34.0 Gbits/sec                  receiver
CPU Utilization: local/sender 99.2% (0.6%u/98.6%s), remote/receiver 12.8% (0.2%u/12.6%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.

Baseline with Flannel

10.141.170.36:

# Terminal 1
sudo docker run -it --entrypoint /bin/bash --name iperf1 networkstatic/iperf3
ip addr show  # 172.30.64.3
iperf3 -s
# Terminal 2
sudo docker run -it --entrypoint /bin/bash --name iperf2 networkstatic/iperf3
ip addr show  # 172.30.64.6
iperf3 -c 172.30.64.3 -V
Linux 9176a9ad64eb 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 GNU/Linux
Time: Thu, 14 Jun 2018 09:21:39 GMT
Connecting to host 172.30.64.3, port 5201
      Cookie: 9176a9ad64eb.1528968099.694202.489a0
      TCP MSS: 1398 (default)
[  4] local 172.30.64.6 port 52567 connected to 172.30.64.3 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.00 GBytes  25.8 Gbits/sec    1    392 KBytes
[  4]   1.00-2.00   sec  2.76 GBytes  23.7 Gbits/sec    0    392 KBytes
[  4]   2.00-3.00   sec  2.99 GBytes  25.7 Gbits/sec    0    400 KBytes
[  4]   3.00-4.00   sec  3.21 GBytes  27.6 Gbits/sec    0    422 KBytes
[  4]   4.00-5.00   sec  3.01 GBytes  25.8 Gbits/sec  184    430 KBytes
[  4]   5.00-6.00   sec  3.06 GBytes  26.3 Gbits/sec    0    430 KBytes
[  4]   6.00-7.00   sec  3.04 GBytes  26.1 Gbits/sec    0    430 KBytes
[  4]   7.00-8.00   sec  3.10 GBytes  26.6 Gbits/sec   93    431 KBytes
[  4]   8.00-9.00   sec  3.05 GBytes  26.2 Gbits/sec    0    437 KBytes
[  4]   9.00-10.00  sec  3.02 GBytes  25.9 Gbits/sec    0    437 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  30.2 GBytes  26.0 Gbits/sec  278             sender
[  4]   0.00-10.00  sec  30.2 GBytes  26.0 Gbits/sec                  receiver
CPU Utilization: local/sender 99.5% (1.0%u/98.5%s), remote/receiver 25.5% (0.9%u/24.7%s)

iperf Done.

FreeFlow

10.141.186.118:

# Terminal 1:
sudo docker run -d -it --privileged --net=host -v /freeflow:/freeflow -e "HOST_IP_PREFIX=10.141.184.0/21" --name freeflow freeflow/freeflow:tcp

sudo docker run -it --entrypoint /bin/bash -v /freeflow:/freeflow -e "VNET_PREFIX=172.30.0.0/16" -e "LD_PRELOAD=/freeflow/libfsocket.so" --name iperf1 networkstatic/iperf3

ip addr show  # 172.30.108.13
iperf3 -s
# Terminal 2:
sudo docker run -it --entrypoint /bin/bash -v /freeflow:/freeflow -e "VNET_PREFIX=172.30.0.0/16" -e "LD_PRELOAD=/freeflow/libfsocket.so" --name iperf2 networkstatic/iperf3

ip addr show  #  172.30.108.12

iperf3 -c 172.30.108.13 -V
Linux c85e8d20aeb9 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 GNU/Linux
Time: Thu, 14 Jun 2018 09:12:14 GMT
Connecting to host 172.30.108.13, port 5201
      Cookie: c85e8d20aeb9.1528967534.283705.49449
      TCP MSS: 1398 (default)
[  5] local 172.30.108.12 port 48248 connected to 172.30.108.13 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  5]   0.00-1.00   sec  4.40 GBytes  37.7 Gbits/sec    0   13.7 KBytes
[  5]   1.00-2.00   sec  2.73 GBytes  23.4 Gbits/sec    0   13.7 KBytes
[  5]   2.00-3.00   sec  3.95 GBytes  33.9 Gbits/sec    0   13.7 KBytes
[  5]   3.00-4.00   sec  4.46 GBytes  38.3 Gbits/sec    0   13.7 KBytes
[  5]   4.00-5.00   sec  4.44 GBytes  38.1 Gbits/sec    0   13.7 KBytes
[  5]   5.00-6.00   sec  4.45 GBytes  38.3 Gbits/sec    0   13.7 KBytes
[  5]   6.00-7.00   sec  3.79 GBytes  32.6 Gbits/sec    0   13.7 KBytes
[  5]   7.00-8.00   sec  3.12 GBytes  26.8 Gbits/sec    0   13.7 KBytes
[  5]   8.00-9.00   sec  4.37 GBytes  37.5 Gbits/sec    0   13.7 KBytes
[  5]   9.00-10.00  sec  4.47 GBytes  38.4 Gbits/sec    0   13.7 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-10.00  sec  40.2 GBytes  34.5 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  40.2 GBytes  34.5 Gbits/sec                  receiver
CPU Utilization: local/sender 99.6% (1.2%u/98.4%s), remote/receiver 56.7% (2.4%u/54.3%s)

iperf Done.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment