Verify api access on CNI DEL for Multus

README.md

Create a net-attach-def.

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bridge-conf
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "mynet",
    "type": "bridge",
    "bridge": "testnet0",
    "ipam": {
        "type": "host-local",
        "subnet": "10.10.0.0/16"
    }
}'
EOF

Go and oc debug node/foo, then chroot /host and create a static pod

cat <<EOT >> /tmp/foo.yaml
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: bridge-conf
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOT

Atomically move it.

mv /tmp/foo.yaml /etc/kubernetes/manifests/

Keep a copy of the original multus kubeconfig

cp /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig /tmp/original.multus.kubeconfig

Change the server multus is using:

sed -i "/server/c\    server: https://[192.0.2.1]:6443" /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig

Verify that it took, you should see the 192.0.2.1 address listed

cat /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig | grep server

Create iptables rule, to be thorough.

iptables -A OUTPUT -d 192.0.2.1 -j DROP

# Or delete it if you have to.
iptables -L OUTPUT --line-numbers
iptables -D OUTPUT 4

Then remove the static pod file...

mv /etc/kubernetes/manifests/foo.yaml /tmp/

This results in a:

  Warning  FailedKillPod  25s    kubelet  error killing pod: failed to "KillPodSandbox" for "84a4e2e73add763cc229dfb79945961e" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l_default_84a4e2e73add763cc229dfb79945961e_0(5ef264ffce29604dcb01efe27f559830e35f26632ca87a2f035b57e56c4fcd0a): error removing pod default_samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2022-04-28T16:00:04Z [error] Multus: [default/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l/84a4e2e73add763cc229dfb79945961e]: error getting pod: Get \\\"https://[192.0.2.1]:6443/api/v1/namespaces/default/pods/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l?timeout=1m0s\\\": dial tcp 192.0.2.1:6443: i/o timeout\\n2022-04-28T16:00:04Z [error] Multus: getPod failed: Multus: [default/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l/84a4e2e73add763cc229dfb79945961e]: error getting pod: Get \\\"https://[192.0.2.1]:6443/api/v1/namespaces/default/pods/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l?timeout=1m0s\\\": dial tcp 192.0.2.1:6443: i/o timeout, but continue to delete\\n\""

But the pod isn't yet getting deleted :/ Darn.