Multus CNI with k3s and RKE

01-multus-k3s.md

Using Multus CNI in K3S

By default, K3S will run with flannel as the CNI and use custom directories to store CNI plugin binaries and config files(You can inspect the kubelet args K3S uses via journalctl -u k3s|grep cni-conf-dir). So you need to configure that properly When deploying Multus CNI.

For example given the official Multus manifests in https://github.com/intel/multus-cni/blob/36f2fd64e0965e639a0f1d17ab754f0130951aba/images/multus-daemonset.yml, the following changes are needed:

volumes:
  - name: cni
    hostPath:
      path: /var/lib/rancher/k3s/agent/etc/cni/net.d
  - name: cnibin
    hostPath:
      path: /var/lib/rancher/k3s/data/<replace-with-your-hash>/bin

containers:
  - name: kube-multus
    image: nfvpe/multus:v3.4.1
    command: ["/entrypoint.sh"]
    args:
      - "--multus-conf-file=auto"
      - "--cni-version=0.3.1"
      # Add the following arg
      - "--multus-kubeconfig-file-host=/var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig"

Once Multus CNI is deployed properly it works as normal in K3S.

02-multus-rke.md

Using Multus CNI in RKE

1. Provision RKE cluster with automatic CNI deployment disabled

RKE cluster.yaml:

...
network:
    plugin: none
services:
  kubelet:
    extra_args:
      cni-conf-dir: "/etc/cni/net.d"
      cni-bin-dir: "/opt/cni/bin"
...

2. Install Multus

Use the following Helm chart for conveniance:

https://github.com/ibrokethecloud/multus-helm

https://github.com/ibrokethecloud/cni-plugins https://ubuntu.com/kubernetes/docs/cni-sriov

03-external.md

• k8snetworkplumbingwg/multus-cni#539
• https://github.com/kubernetes-sigs/kubespray/blob/master/docs/multus.md
• https://medium.com/@sarpkoksal/multus-installation-on-kubernetes-a0a6ef04972f
• https://github.com/k8snetworkplumbingwg/reference-deployment/blob/a3b4d3066ac0e130deee7d20fa02a72d67e9e683/multus-calico/multus-crio-daemonset.yml
• https://github.com/rancher/kontainer-driver-metadata/blob/497e5d720660bf9c8dc7ab0c7b44ee6daca28534/rke/templates/calico.go
• rancher/rke#873 (comment)

I believe you can replace <replace-with-your-hash> with current to simplify this.

Hey, Ive been able to install MultusCNI with your solution and use macvlan. I have a net-attach with the spec inside. I've been trying to use the NetworkAttachmentDefinition with CNI config file instead of the spec inside the yaml file and I cant make it work. I keep getting this error:

Multu ││ s: [max-dev/mc-file2]: error loading k8s delegates k8s args: TryLoadPodDelegates: error in getting k8s network for p ││ od: GetNetworkDelegates: failed getting the delegate: GetCNIConfig: err in GetCNIConfigFromFile: No networks found in /etc/cni/multus/net.d                     

Any idea on how to solve this? Thank you!