Docker container network timeout on private IPs over VPN

docker_network_timeout_ips_vpn.md

I ran into a weird issue while setting up a docker image:

• the image was based on alpine and has kubectl and awscli installed
• both aws credentials (~/.aws/credentials) and kube config(~/.kube/config) are mounted to the container
  • the command for docker looks like: docker run -it --rm -v ${HOME}/.aws:/root/.aws -v ${HOME}/.kube:/root/.kube <my_image> /bin/bash
• my laptop is connected to VPN and my k8s cluster is on a VPC accessible only via VPN.

What I observed:

• on the host (my laptop):
  • both aws and kubectl calls are successful.
• in the container:
  • aws calls are successful. e.g. aws s3 ls s3://my-bucket
  • kubectl calls are timed out consistently: k8s get pod -A

Added --v=8 to kubectl call (on both host and contaienr): the right endpoint is used for both host and container.

Tried nslookup on both host and container for the k8s endpoint and the private IPs are resolved correctly.

Tried telnet <private_ip> 443 on both host and container and saw the same thing:

• the host was able to establish a connection
• the container stuck on establishing a connection

Started tcpdump (in a different shell, sudo tcpdump -nnA -s 0 "host <private_ip>") on both host and container to view traffic while command (either telnetorkubectl`) was executed in the container:

• tcpdump in the container showed traffic
• tcpdump on the host did not show any traffic

Checked route table in the container (netstat -nr):

bash-4.4# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.17.0.1      0.0.0.0         UG        0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0

the private ips my k8s cluster resovled to do not fall into the 172.17 band so there should not be any collision.

At this point, the docker run options seem correct and network configuration seems fine too.

After a bit more research, I found out this is a known issue for docker on Mac OS X, and the solution is a simple docker network prune.

I hope this would be helpful to whoever ran into this issue