TCP Simultaneous Open & connect to itself

tcp_tso_self_connect.md

A realworld example showing how mixing ephemeral port range with server listen address can lead to port conflicts (due to unintuitive TCP Simulataneous Open (TSO) on same host, see RFC793)

In this example, we will run a "webserver" and a "health check client" in the same machine

1. Run this on Terminal1 (this is "webserver"):

$ while echo -e "HTTP/1.0 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 6\r\n\r\nhello" | \
    socat -t 0.1 TCP-L:33000,reuseaddr STDIN; do echo "Served request"; done

2. Run this one Terminal2:

$ while true; do socat TCP4:127.0.0.1:33000 STDOUT; [ $? -eq 130 ] && break; done

Make sure they are working

3. Now in Terminal1, press ctrl-c This will make the "webserver" exit and make the "client" go into loop until it connects to itself.

4. In Terminal2, wait for client to connect to itself (aka wait for the scrolling console output to pause, might take sometime)

   Finally it would look something like this:

....
2020/01/28 13:23:37 socat[683610] E connect(5, AF=2 127.0.0.1:33000, 16): Connection refused
2020/01/28 13:23:37 socat[683611] E connect(5, AF=2 127.0.0.1:33000, 16): Connection refused
...output paused...

5. In new terminal, confirm that it is connected to itself

$ sudo netstat -atnp | grep 33000
tcp        0      0 127.0.0.1:33000         127.0.0.1:33000         ESTABLISHED 683612/socat

6. In Terminal1, re-run the socat http server command, you should get something like below

$ while echo -e "HTTP/1.0 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 6\r\n\r\nhello" | \
    socat -t 0.1 TCP-L:33000,reuseaddr STDIN; do echo "Served request"; done
2020/01/28 13:26:39 socat[685692] E bind(5, {AF=2 0.0.0.0:33000}, 16): Address already in use 

7. To fix

   7.1 Don't use server listen ports from ephemeral range

   $ sysctl net.ipv4.ip_local_port_range
   net.ipv4.ip_local_port_range = 32768    60999

   7.2 Or use SO_REUSEADDR/SO_REUSEPORT option

   So above commands become:

   Terminal1

   $ while echo -e "HTTP/1.0 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 6\r\n\r\nhello" | \
        socat -t 0.1 TCP-L:33000,reuseaddr,reuseport STDIN; do echo "Served request"; done

   Terminal2

   $ while true; do socat TCP4:127.0.0.1:33000,reuseaddr,reuseport STDOUT; [ $? -eq 130 ] && break; done   

   Now it should look like this when we reproduce the problem:

   $ sudo netstat -atnp | grep 33000
   tcp        0      0 0.0.0.0:33000           0.0.0.0:*               LISTEN      763427/socat
   tcp        0      0 127.0.0.1:33000         127.0.0.1:33000         ESTABLISHED 763336/socat