Multus CNI with k3s and RKE

01-multus-k3s.md

Using Multus CNI in K3S

By default, K3S will run with flannel as the CNI and use custom directories to store CNI plugin binaries and config files(You can inspect the kubelet args K3S uses via journalctl -u k3s|grep cni-conf-dir). So you need to configure that properly When deploying Multus CNI.

For example given the official Multus manifests in https://github.com/intel/multus-cni/blob/36f2fd64e0965e639a0f1d17ab754f0130951aba/images/multus-daemonset.yml, the following changes are needed:

volumes:
  - name: cni
    hostPath:
      path: /var/lib/rancher/k3s/agent/etc/cni/net.d
  - name: cnibin
    hostPath:
      path: /var/lib/rancher/k3s/data/<replace-with-your-hash>/bin

containers:
  - name: kube-multus
    image: nfvpe/multus:v3.4.1
    command: ["/entrypoint.sh"]
    args:
      - "--multus-conf-file=auto"
      - "--cni-version=0.3.1"
      # Add the following arg
      - "--multus-kubeconfig-file-host=/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig"

Once Multus CNI is deployed properly it works as normal in K3S.

02-multus-rke.md

Using Multus CNI in RKE

1. Provision RKE cluster with automatic CNI deployment disabled

RKE cluster.yaml:

...
network:
    plugin: none
services:
  kubelet:
    extra_args:
      cni-conf-dir: "/etc/cni/net.d"
      cni-bin-dir: "/opt/cni/bin"
...

2. Install Multus

Use the following Helm chart for conveniance:

https://github.com/ibrokethecloud/multus-helm

https://github.com/ibrokethecloud/cni-plugins https://ubuntu.com/kubernetes/docs/cni-sriov

03-external.md

• k8snetworkplumbingwg/multus-cni#539
• https://github.com/kubernetes-sigs/kubespray/blob/master/docs/multus.md
• https://medium.com/@sarpkoksal/multus-installation-on-kubernetes-a0a6ef04972f
• https://github.com/k8snetworkplumbingwg/reference-deployment/blob/a3b4d3066ac0e130deee7d20fa02a72d67e9e683/multus-calico/multus-crio-daemonset.yml
• https://github.com/rancher/kontainer-driver-metadata/blob/497e5d720660bf9c8dc7ab0c7b44ee6daca28534/rke/templates/calico.go
• rancher/rke#873 (comment)

I believe you can replace <replace-with-your-hash> with current to simplify this.

Hey, Ive been able to install MultusCNI with your solution and use macvlan. I have a net-attach with the spec inside. I've been trying to use the NetworkAttachmentDefinition with CNI config file instead of the spec inside the yaml file and I cant make it work. I keep getting this error:

Multu ││ s: [max-dev/mc-file2]: error loading k8s delegates k8s args: TryLoadPodDelegates: error in getting k8s network for p ││ od: GetNetworkDelegates: failed getting the delegate: GetCNIConfig: err in GetCNIConfigFromFile: No networks found in /etc/cni/multus/net.d                     

Any idea on how to solve this? Thank you!

I too ran into some issues installing multus on k3s (although in my case, on a Raspberry Pi 4, not RKE). This post helped point me in the right direction (thank you), so I figured I'd post my learnings here.

Like the original post alludes to, k3s uses non-standard file locations for its CNI configurations and binaries. Multus expects those configurations and binaries to be in the same place. Trying to update the multus DaemonSet to accommodate this was pretty messy, so I instead created a DaemonSet that symlinks the k3s file locations to the standard ones, and this allows the standard multus DaemonSet to run correctly.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: cni-symlinker
  name: cni-symlinker
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: cni-symlinker
  template:
    metadata:
      labels:
        app: cni-symlinker
    spec:
      nodeSelector:
        # Ensure this ONLY runs on k3s instances
        node.kubernetes.io/instance-type: k3s
      initContainers:
        - name: cni-symlinker
          image: busybox
          command:
            - /bin/sh
          args:
            - -c
            - |
              if [ ! -L /host/etc/cni/net.d ]; then
                  ln -s /var/lib/rancher/k3s/agent/etc/cni/net.d /host/etc/cni/net.d
              fi
              if [ ! -L /host/opt/cni/bin ]; then
                  ln -s /var/lib/rancher/k3s/data/current/bin /host/opt/cni/bin
              fi
          securityContext:
            privileged: true
          volumeMounts:
            - name: etc
              mountPath: /host/etc/cni
            - name: opt
              mountPath: /host/opt/cni
      # Need to sleep forever otherwise DaemonSet won't be healthy
      containers:
        - name: sleep-forever
          image: busybox
          command:
            - sleep
          args:
            - infinity
      volumes:
        - name: etc
          hostPath:
            path: /etc/cni
        - name: opt
          hostPath:
            path: /opt/cni

Keep in mind, you must run this DaemonSet BEFORE installing multus, otherwise multus will create actual directories where the symlinks would be. If you already ran multus prior, you can clean up your host nodes with:

rm -r /etc/cni/net.d /opt/cni/bin

WARNING: if the above commands are executed in the incorrect context, they may completely hose your Kubernetes nodes. The same goes for multus misconfiguration in general.

Also, I've recently started using ArgoCD to deploy multus - you can check out the WIP here. It works well, but I am currently changing my argocd configuration frequently, so ymmv.

Hello is there a new technique to deploy multus cni on k3s Am stuck with error

Hello is there a new technique to deploy multus cni on k3s Am stuck with error

this is what I did that worked for me. adding the clusterNetwork line made it work. before that I manually installed flannel over the built-in version. Created a new test cluster and ran this helmchart like this and it worked fine.

# k3s multus install
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: multus
  namespace: kube-system
spec:
  repo: https://rke2-charts.rancher.io
  chart: rke2-multus
  targetNamespace: kube-system
  # createNamespace: true
  valuesContent: |-
    config:
      cni_conf:
        confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
        clusterNetwork: /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist
        binDir: /var/lib/rancher/k3s/data/current/bin/
        kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig