It's very useful to understand the bandwidth, throughput and latency of the remote computer you're trying to communicate with. It can inform you how well you can play a multiplayer game (latency), or the bitrate at which you can stream a video (throughput).
The difference between bandwidth and throughput is:
- Bandwidth : Refers how fast a device can send data over a single communication channel .
- Throughput: Refers how fast a device is actually sending data over the communication channel.
I think networked applications should tell you up front the recommended latency and bandwidth requirements for adequate usage, but many don't! Think of it like a requirement list on game boxes or ingredient list on food packaging. However I have found lists such as:
Let's call our computer "Node A", and the remote computer to stream to "Node B". Let us also create a little function in case you're runnin Bash:
$ '
repeat - Repeatedly run a command. Basically the same as ZSH \`repeat\`.
Usage: repeat <times> <command>
'
repeat () {
times="$1"
shift
seq "$times" | xargs -I -- $*
}For ZSH, there's already a builtin repeat.
The most accurate test would be directly use iperf between Node A and Node B. There are currently 2 versions of iperf in use, version 2 and version 3. They are not compatible with each other. For now I'll be using iperf2. But the same test can be done in iperf3 withs slightly different flags.
Here's our first TCP test, we won't tune TCP at all, and leave it everything as default. We are going to setup Node A as the server, and Node B as the client. However, through the use of the --tradeoff flag both Nodes will be acting as a server and client. The --tradeoff flag will allow us to find the throughput for both upload and download between Node A and Node B.
The tests alternate between Node B sending data to Node A, and then Node A sending data to Node B. You'll see this in the IP addresses and the port allocations. And because network testing will always encounter noise and artifacts, we'll repeat this 10 times.
Note at the moment, iperf3 has not implemented the --tradeoff flag, so you just need to it once without the --reverse flag and once with in iperf3.
On Node A:
> iperf --server --port 55555 --num 100M
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50310
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-14.6 sec 100 MBytes 57.5 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.6.235 port 33065 connected with 10.20.1.111 port 55555
[ 4] 0.0-11.9 sec 100 MBytes 70.4 Mbits/sec
[ 5] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50311
[ 5] 0.0-14.2 sec 100 MBytes 59.2 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 5] local 10.20.6.235 port 33066 connected with 10.20.1.111 port 55555
[ 5] 0.0-12.1 sec 100 MBytes 69.2 Mbits/sec
[ 4] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50312
[ 4] 0.0-14.3 sec 100 MBytes 58.5 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 162 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.6.235 port 33067 connected with 10.20.1.111 port 55555
[ 4] 0.0-12.9 sec 100 MBytes 65.2 Mbits/sec
[ 5] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50314
[ 5] 0.0-14.2 sec 100 MBytes 59.2 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 5] local 10.20.6.235 port 33068 connected with 10.20.1.111 port 55555
[ 5] 0.0-11.7 sec 100 MBytes 71.4 Mbits/sec
[ 4] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50316
[ 4] 0.0-14.4 sec 100 MBytes 58.4 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 119 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.6.235 port 33069 connected with 10.20.1.111 port 55555
[ 4] 0.0-12.5 sec 100 MBytes 67.2 Mbits/sec
[ 5] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50319
[ 5] 0.0-13.8 sec 100 MBytes 60.7 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 5] local 10.20.6.235 port 33070 connected with 10.20.1.111 port 55555
[ 5] 0.0-11.8 sec 100 MBytes 71.1 Mbits/sec
[ 4] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50329
[ 4] 0.0-14.1 sec 100 MBytes 59.7 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.6.235 port 33071 connected with 10.20.1.111 port 55555
[ 4] 0.0-11.9 sec 100 MBytes 70.7 Mbits/sec
[ 5] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50331
[ 5] 0.0-16.6 sec 100 MBytes 50.5 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 5] local 10.20.6.235 port 33072 connected with 10.20.1.111 port 55555
[ 5] 0.0-12.1 sec 100 MBytes 69.5 Mbits/sec
[ 4] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50332
[ 4] 0.0-13.8 sec 100 MBytes 60.7 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.6.235 port 33073 connected with 10.20.1.111 port 55555
[ 4] 0.0-11.9 sec 100 MBytes 70.5 Mbits/sec
[ 5] local 10.20.6.235 port 55555 connected with 10.20.1.111 port 50333
[ 5] 0.0-15.4 sec 100 MBytes 54.3 Mbits/sec
------------------------------------------------------------
Client connecting to 10.20.1.111, TCP port 55555
TCP window size: 144 KByte (default)
------------------------------------------------------------
[ 5] local 10.20.6.235 port 33074 connected with 10.20.1.111 port 55555
[ 5] 0.0-11.9 sec 100 MBytes 70.7 Mbits/sec
On Node B:
> repeat 10 iperf --client 10.20.6.235 --port 55555 --num 100M --tradeoff
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.20.1.111 port 50310 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-14.6 sec 100 MBytes 57.5 Mbits/sec
[ 3] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33065
[ 3] 0.0-11.9 sec 100 MBytes 70.2 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.1.111 port 50311 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-14.2 sec 100 MBytes 59.2 Mbits/sec
[ 4] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33066
[ 4] 0.0-12.2 sec 100 MBytes 68.9 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.1.111 port 50312 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-14.3 sec 100 MBytes 58.5 Mbits/sec
[ 4] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33067
[ 4] 0.0-12.9 sec 100 MBytes 65.0 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.20.1.111 port 50314 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-14.2 sec 100 MBytes 59.2 Mbits/sec
[ 3] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33068
[ 3] 0.0-11.8 sec 100 MBytes 71.2 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.1.111 port 50316 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-14.3 sec 100 MBytes 58.5 Mbits/sec
[ 4] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33069
[ 4] 0.0-12.5 sec 100 MBytes 67.1 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.1.111 port 50319 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-13.8 sec 100 MBytes 60.7 Mbits/sec
[ 4] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33070
[ 4] 0.0-11.8 sec 100 MBytes 70.9 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.1.111 port 50329 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-14.0 sec 100 MBytes 59.7 Mbits/sec
[ 4] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33071
[ 4] 0.0-11.9 sec 100 MBytes 70.4 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.20.1.111 port 50331 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-16.6 sec 100 MBytes 50.5 Mbits/sec
[ 3] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33072
[ 3] 0.0-12.1 sec 100 MBytes 69.3 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.1.111 port 50332 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-13.8 sec 100 MBytes 60.7 Mbits/sec
[ 4] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33073
[ 4] 0.0-11.9 sec 100 MBytes 70.3 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.20.6.235, TCP port 55555
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.20.1.111 port 50333 connected with 10.20.6.235 port 55555
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-15.4 sec 100 MBytes 54.3 Mbits/sec
[ 4] local 10.20.1.111 port 55555 connected with 10.20.6.235 port 33074
[ 4] 0.0-11.9 sec 100 MBytes 70.5 Mbits/sec
On iperf2, there's a report to CSV option, while iperf3 has the option to report in JSON. So if we want to calculate averages and quartiles... etc, make sure to use those formats and a tool like json2csv.
If we cannot use iperf because we don't have control over both Node A and Node B, the next best thing is to use http://speedtest.net/. The trick is to select a server to test against, that is geographically and economically close to the computers you intend to stream to. By economically, I mean if you can find the a server ran by the same provider as their internet service provider, then that's even better.
Note that if know that Node A has a high bandwidth connection, make sure the server you're testing against with speedtest.net can match or be greater than your connection. This is unlikely for most usecases, as those servers on speedtest.net are ran by large internet service providers.
We can either use the website http://speedtest.net/ or we can use speedtest-cli. Note that there can be discrepancies between the 2, as the website uses Flash and pure sockets, while speedtest-cli uses Python and HTTP.
> speedtest-cli --list | less # acquire server id from an appropriate server
> speedtest-cli --server $SERVER_ID # enter the server id found
Also connections can be quite variable over time and over distance, so do a couple of tests to remove outliers.
> # if you're in ZSH
> repeat 10 speedtest-cli --server $SERVER_ID --simple | tee /tmp/speedtest_stats
> # if you're in Bash
> seq 10 | xargs -I -- speedtest-cli --server $SERVER_ID --simple | tee /tmp/speedtest_stats
Here's the data I got for running speedtest-cli on a geographically close server:
Ping: 3.48 ms
Download: 212.45 Mbits/s
Upload: 132.20 Mbits/s
Ping: 3.288 ms
Download: 211.08 Mbits/s
Upload: 123.81 Mbits/s
Ping: 3.323 ms
Download: 234.33 Mbits/s
Upload: 128.56 Mbits/s
Ping: 3.492 ms
Download: 220.14 Mbits/s
Upload: 128.54 Mbits/s
Ping: 3.528 ms
Download: 201.08 Mbits/s
Upload: 125.25 Mbits/s
Ping: 3.541 ms
Download: 161.70 Mbits/s
Upload: 128.58 Mbits/s
Ping: 3.558 ms
Download: 213.32 Mbits/s
Upload: 132.04 Mbits/s
Ping: 3.649 ms
Download: 215.76 Mbits/s
Upload: 126.88 Mbits/s
Ping: 3.704 ms
Download: 195.75 Mbits/s
Upload: 128.62 Mbits/s
Ping: 3.177 ms
Download: 190.49 Mbits/s
Upload: 118.22 Mbits/s
Now to calculate the average using grep, perl and R:
> cat /tmp/speedtest_stats | grep 'Ping' | perl -p -e 's/.*?(\d*\.\d*).*/\1/' | Rscript -e 'print(summary(scan("stdin")))'
Read 10 items
Min. 1st Qu. Median Mean 3rd Qu. Max.
3.177 3.362 3.510 3.474 3.554 3.704
> cat /tmp/speedtest_stats | grep 'Download' | perl -p -e 's/.*?(\d*\.\d*).*/\1/' | Rscript -e 'print(summary(scan("stdin")))'
Read 10 items
Min. 1st Qu. Median Mean 3rd Qu. Max.
161.7 197.1 211.8 205.6 215.2 234.3
> cat /tmp/speedtest_stats | grep 'Upload' | perl -p -e 's/.*?(\d*\.\d*).*/\1/' | Rscript -e 'print(summary(scan("stdin")))'
Read 10 items
Min. 1st Qu. Median Mean 3rd Qu. Max.
118.2 125.7 128.6 127.3 128.6 132.2
Once you know your network capabilities, it would be preferable to get Node B to also do a speed test to a server that is geographically and economically close to Node A. This because even if Node A can upload 10 Mb/s to a speedtest.net server or Node B, if Node B can only download 5 Mb/s , then the final data transfer will only occur at 5 Mb/s.
One of things you may notice is that your bandwidth gets lower as you select servers farther away from your geographic location. I don't fully understand why this happens, but it seems it has something to do with TCP window size, and other TCP tunables.
Downloading from a local server is generally faster than a distant one. This is mostly due to the fact that parameters (such as the TCP window size) controlling the transfer are not optimized for the increased latency that comes from an increase in distance. Each data packet sent by a server must be acknowledged as received before another one can be sent. While this feature makes sure that data arrives "bit perfect," the greater latency created by distance alone can have a significant effect -- especially with larger file transfers, as more packets means greater delay. https://support.speedtest.net/hc/en-us/articles/203845640-Why-does-Speedtest-net-recommend-a-nearby-server-
There are a variety of ways to tune TCP for high latency connections, one is to use a download manager that downloads multiple parts of a file all at once, or change the TCP receive memory or window size, so you can have more data on the line at once, this will increase the bandwidth of a single TCP session without multithreading. The first option is often used by non-technical people, or people who cannot fully control their computer. The second option is often used by system administrators and server operators.
For more information, see: https://en.wikipedia.org/wiki/TCP_tuning & http://www.kehlet.cx/articles/99.html
By the way, on the topic of servers and geography, there's a package called geoip-tools or geoip or geoip-bin, that allows you look up the geographic location of a hostname or IP address. This can also be useful if all you have of Node B is their IP address and nothing else. Use it like geoiplookup 8.8.8.8. Then you can use the City or Country and used speedtest-cli for those countries/cities. For more information, see: http://kbeezie.com/geoiplookup-command-line/
It would be an interesting exercise to do a speed test on every country and major city from where currently are, this could give you a map of latencies and bandwidth from your current location. Then visualise onto a world map. Note that speedtest.net does have API limits, so becareful.