Skip to content

Instantly share code, notes, and snippets.

@jaytaylor

jaytaylor/k8s-gigawatt-messed-up-subnetting-networking-workaround.md

Last active February 24, 2018 00:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jaytaylor/942abb66f8fa30d26a3e7f565140e426 to your computer and use it in GitHub Desktop.
Save jaytaylor/942abb66f8fa30d26a3e7f565140e426 to your computer and use it in GitHub Desktop.
Download ZIP
Kubernetes kubeadm-setup.sh: Workaround for Gigawatt's crazy DC networking..
Raw
k8s-gigawatt-messed-up-subnetting-networking-workaround.md

Initial symptoms

Kubernetes master setup went fine, but when I tried to join a kbminion slave node:

Checking whether docker can run container ...
Checking iptables default rule ...
Checking br_netfilter module ...
Checking sysctl variables ...
Check successful, ready to run 'join' command ...
[preflight] Skipping pre-flight checks
[validation] WARNING: kubeadm doesn't fully support multiple API Servers yet
[discovery] Trying to connect to API Server "10.196.40.225:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.196.40.225:6443"
[discovery] Trying to connect to API Server "10.196.40.225:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.196.40.225:6443"
[discovery] Failed to request cluster info, will try again: [Get https://10.196.40.225:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.196.40.225:6443: getsockopt: no route to host]
[discovery] Failed to request cluster info, will try again: [Get https://10.196.40.225:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.196.40.225:6443: getsockopt: no route to host]
[discovery] Failed to request cluster info, will try again: [Get https://10.196.40.225:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.196.40.225:6443: getsockopt: no route to host]
[discovery] Failed to request cluster info, will try again: [Get https://10.196.40.225:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.196.40.225:6443: getsockopt: no route to host]
[discovery] Failed to request cluster info, will try again: [Get https://10.196.40.225:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 10.196.40.225:6443: getsockopt: no route to host]
...

Searching for kubernetes "Failed to request cluster info, will try again:" "6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp" "getsockopt: no route to host" yielded Failed to request cluster info - getsockopt: no route to host, but ultimately was the wrong path.

Firewall check

Service totally disabled in systemd.

Pods check on master

There was an unhappy DNS pod, which was easily fixed by pulling a missing docker image, tagging it, and pushing to the local repo.

Eventually I discovered an oddity..

ip addr on the master

jay@gwdc01ppi:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether fa:16:3e:7b:ed:f9 brd ff:ff:ff:ff:ff:ff
    inet 10.196.40.225/20 brd 10.196.47.255 scope global dynamic eth0
       valid_lft 80677sec preferred_lft 80677sec   ######################################
    inet 10.242.243.25/32 scope global eth0        # <-- NB: This is a "fake" interface #
       valid_lft forever preferred_lft forever     #         I added, just ignore it.   #
    inet6 fe80::f816:3eff:fe7b:edf9/64 scope link  #         (It does nothing AFAICT)   #
       valid_lft forever preferred_lft forever     ######################################
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:e1:ac:89:55 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:e1ff:feac:8955/64 scope link
       valid_lft forever preferred_lft forever
5: veth77457a6@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP
    link/ether 0a:9c:2f:53:4d:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::89c:2fff:fe53:4de4/64 scope link
       valid_lft forever preferred_lft forever
28: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    link/ether 7a:97:9e:99:9b:b3 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::7897:9eff:fe99:9bb3/64 scope link
       valid_lft forever preferred_lft forever
29: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
    link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::6400:3bff:fe35:e68f/64 scope link
       valid_lft forever preferred_lft forever
30: vethf60bc09c@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP
    link/ether 9a:9b:c9:0c:6b:35 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::989b:c9ff:fe0c:6b35/64 scope link
       valid_lft forever preferred_lft forever
31: vethce8986b6@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP
    link/ether 7e:13:6d:25:8c:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::7c13:6dff:fe25:8cc3/64 scope link
       valid_lft forever preferred_lft forever

Okay, so it thinks its IP is 10.196.40.225, cool..

From a kbminion slave node:

root@gwdc01epb:~# ping gwdc01ppi
PING gwdc01ppi (10.242.243.25) 56(84) bytes of data.
64 bytes from gwdc01ppi (10.242.243.25): icmp_seq=1 ttl=55 time=1.04 ms
64 bytes from gwdc01ppi (10.242.243.25): icmp_seq=2 ttl=55 time=1.08 ms
^C
--- gwdc01ppi ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.040/1.062/1.084/0.022 ms

Wait, what? The slave thinks the master's IP is 10.242.243.25.

Furthermore, the slave can't even each the master on 10.196.40.225.

I've been fighting this all day (and most of another day previously), and after many failures have finally figured out a workaround.

The solution is to add an iptables rule on the slaves to forward traffic bound for 10.196.40.225 -> 10.242.243.25.

On all the slaves:

iptables -t nat -A OUTPUT -p tcp -d 10.196.40.225 --dport 6443 -j DNAT --to-destination 10.242.243.25:6443

Source: I found this gem here.

Making it permanent

sudo chmod a+x /etc/rc.local
echo '/sbin/iptables -t nat -A OUTPUT -p tcp -d 10.196.40.225 --dport 6443 -j DNAT --to-destination 10.242.243.25:6443' \
    | sudo tee -a /etc/rc.local

Update 2018-02-23

Don't forget to perform the reverse procedure for communication from the master to the kbminion slave nodes..

Example

Determine aliased IP from master:

kbmaster:~$ ping slave1

Obtain eth0 IP on slave1:

kbslave01:~$ ip addr

Also see: https://gist.github.com/jaytaylor/ddcc88cb4e3c2f260c7802414f85c43b.

Enjoy!

Footnotes

Things that either weren't fully explored or didn't pan out:

Additional resources discovered:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment