Create a net-attach-def.
cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: bridge-conf
spec:
config: '{
"cniVersion": "0.3.1",
"name": "mynet",
"type": "bridge",
"bridge": "testnet0",
"ipam": {
"type": "host-local",
"subnet": "10.10.0.0/16"
}
}'
EOF
Go and oc debug node/foo
, then chroot /host
and create a static pod
cat <<EOT >> /tmp/foo.yaml
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: bridge-conf
spec:
containers:
- name: samplepod
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
EOT
Atomically move it.
mv /tmp/foo.yaml /etc/kubernetes/manifests/
Keep a copy of the original multus kubeconfig
cp /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig /tmp/original.multus.kubeconfig
Change the server multus is using:
sed -i "/server/c\ server: https://[192.0.2.1]:6443" /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig
Verify that it took, you should see the 192.0.2.1 address listed
cat /etc/kubernetes/cni/net.d/multus.d/multus.kubeconfig | grep server
Create iptables rule, to be thorough.
iptables -A OUTPUT -d 192.0.2.1 -j DROP
# Or delete it if you have to.
iptables -L OUTPUT --line-numbers
iptables -D OUTPUT 4
Then remove the static pod file...
mv /etc/kubernetes/manifests/foo.yaml /tmp/
This results in a:
Warning FailedKillPod 25s kubelet error killing pod: failed to "KillPodSandbox" for "84a4e2e73add763cc229dfb79945961e" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l_default_84a4e2e73add763cc229dfb79945961e_0(5ef264ffce29604dcb01efe27f559830e35f26632ca87a2f035b57e56c4fcd0a): error removing pod default_samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l from CNI network \"multus-cni-network\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): netplugin failed: \"2022-04-28T16:00:04Z [error] Multus: [default/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l/84a4e2e73add763cc229dfb79945961e]: error getting pod: Get \\\"https://[192.0.2.1]:6443/api/v1/namespaces/default/pods/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l?timeout=1m0s\\\": dial tcp 192.0.2.1:6443: i/o timeout\\n2022-04-28T16:00:04Z [error] Multus: getPod failed: Multus: [default/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l/84a4e2e73add763cc229dfb79945961e]: error getting pod: Get \\\"https://[192.0.2.1]:6443/api/v1/namespaces/default/pods/samplepod-ci-ln-lxbsj1b-72292-77qjp-worker-a-bx48l?timeout=1m0s\\\": dial tcp 192.0.2.1:6443: i/o timeout, but continue to delete\\n\""
But the pod isn't yet getting deleted :/ Darn.