Workaround changing legacy iptables to nf_tables when upgrading Ubuntu 20.04 to Ubuntu 22.04

readme.md

Workaround changing legacy iptables to nf_tables when upgrading Ubuntu 20.04 to Ubuntu 22.04

Problem

When hosts are upgraded from Ubuntu 20.04 to Ubuntu 22.04, the host iptables is changed from legacy to nf_tables.

# Ubuntu 20.04
$ iptables --version
iptables v1.8.4 (legacy)
# Ubuntu 22.04
$ iptables --version
iptables v1.8.7 (nf_tables)

As RKE1 uses Docker containers, the containers created to run the Kubernetes components contain an iptables-wrapper script which determines what iptables mode needs to be used. This script is run after the container is created and sets up the correct symbolic links to the iptables binaries.

# Hyperkube image first start
$ docker exec kubelet ls -la /etc/alternatives/iptables
lrwxrwxrwx 1 root root 26 Jul 17 07:29 /etc/alternatives/iptables -> /usr/sbin/iptables-wrapper                                                                                                                                                                                                                               
# Ubuntu 20.04 (legacy)
$ docker exec kubelet ls -la /etc/alternatives/iptables
lrwxrwxrwx 1 root root 25 Jul 17 07:42 /etc/alternatives/iptables -> /usr/sbin/iptables-legacy
# Ubuntu 22.04 (nf_tables)
$ docker exec kubelet ls -la /etc/alternatives/iptables
lrwxrwxrwx 1 root root 22 Jul 17 08:01 /etc/alternatives/iptables -> /usr/sbin/iptables-nft

The problem that arises after switching iptables mode (what happens when upgrading from Ubuntu 20.04 to Ubuntu 22.04), is that the iptables-wrapper script only gets executed when the container is first started. This means that mode changes do no get applied in existing containers. Even when the container gets re-created, the existence of legacy iptables rules will cause the wrapper script to default to legacy iptables again.

Workaround

To manually switch iptables mode in the Kubernetes containers as a workaround, there are 2 requirements:

• Set the iptables binary to the iptables-wrapper script to let it automatically determine the right mode
• Absence of any legacy iptables rules

Step 0: Prepare the host according to the environment

The Kubernetes containers on the host will go down, so any action needed to prepare the node for maintenance needs to be executed. This depends on each environment, think about cordon/drain and/or other environment specific steps before a host goes down/in maintenance.

Step 1: set the iptables binary to iptables-wrapper

This involves exec-ing into the containers and setting the iptables binary, and immediately stopping the container as whenever the iptables binary is called, it will reset to legacy because legacy iptables rules are present.

In Rancher provisioned clusters, the worker only nodes will execute a nodeplan every interval (default 120 seconds). We need to stop the cattle-node-agent pod container as it will restart the kubelet before the reboot of the host is initiated.

for container in kubelet kube-proxy; do
    docker exec $container update-alternatives --set iptables /usr/sbin/iptables-wrapper && docker stop $container
    if [ "$container" = "kubelet" ]; then
        docker stop $(docker ps | grep cattle-node-agent | awk '{ print $1 }')
    fi
done

Step 2: Clear the legacy iptables rules

The easiest way to do this is to reboot the host.

reboot

Step 3: Monitor the container

Whenever the host is back online, you can monitor the Kubernetes containers to see if the iptables-wrapper was executed properly.

# Correct link
$ docker exec kubelet ls -la /etc/alternatives/iptables
lrwxrwxrwx 1 root root 22 Jul 17 08:01 /etc/alternatives/iptables -> /usr/sbin/iptables-nft
# Correct version
$ docker exec kubelet iptables --version
iptables v1.8.7 (nf_tables)

Other connectivity tests can be performed to ensure everything is working properly.