Testing Multus CNI on Talos Kubernetes

README.md

Talos + Qemu + Multus Dbg Testing

About

The motivation for this gist comes from an interest in reproducing issues observed using multus on talos metal.

Here we will attempt to re-create the symptom in as simple of an environment as possible to then pursue support as needed.

Issue Reproduction

Dependencies

• linux
• qemu
• talosctl
• kubectl

Steps

# download latest talosct
curl -L https://github.com/siderolabs/talos/releases/download/v1.6.7/talosctl-linux-amd64 --output ~/.local/bin/talosctl
chmod +x ~/.local/bin/talosctl
talosctl version

# Create a new qemu driver talos k8s cluster
sudo -E ~/.local/bin/talosctl cluster create \
  --crashdump \
  --name talos-default \
  --kubernetes-version 1.29.3 \
  --provisioner qemu \
  --arch amd64 \
  --controlplanes 1 \
  --memory 4092 \
  --cpus 4 \
  --workers 2 \
  --memory-workers 4092 \
  --cpus-workers 4 \
  --disk 6144 \
  --extra-disks 1 \
  --extra-disks-size 6144 \
  --wait=true --with-debug \
  --with-init-node

### Install Multus
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml

### Check config on nodes
talosctl --talosconfig ~/.talos/config -e 10.5.0.2 -n 10.5.0.3 ls /etc/cni/net.d
talosctl --talosconfig ~/.talos/config -e 10.5.0.2 -n 10.5.0.3 read /etc/cni/net.d/00-multus.conf | jq .
talosctl --talosconfig ~/.talos/config -e 10.5.0.2 -n 10.5.0.3 ls /opt/cni/bin

# List node links
talosctl --talosconfig ~/.talos/config -e 192.168.1.164 -n 192.168.1.164 get link

### Create multus network
cat <<EOF | kubectl apply -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: br0
  namespace: kube-system
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "macvlan",
      "master": "br0",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.1.0/24",
        "rangeStart": "192.168.1.200",
        "rangeEnd": "192.168.1.216",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": "192.168.1.1"
      }
    }'
EOF

### List net-attach-def
kubectl -n kube-system get network-attachment-definitions

### Run test pod
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  namespace: kube-system
  annotations:
    k8s.v1.cni.cncf.io/networks: br0
spec:
  containers:
  - name: test
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine
EOF