ubuntu-20.04 kvm with vfio-passthrough networking

kvm-vfio.md

fix apparmor vim /etc/apparmor.d/abstractions/libvirt-qemu

find the vfio line and add a second (wildcard) line:

 # for vfio hotplug on systems without static vfio (LP: #1775777)
  /dev/vfio/vfio rw,
  /dev/vfio/* rw,

add iommu=pt intel_iommu=on to kernel params

set up vfio in netplan:

network:
  version: 2
  ethernets:
    eno1:
      dhcp4: false
      virtual-function-count: 7
    eno2:
      dhcp4: true
      virtual-function-count: 7
    eno3:
      dhcp4: false
      virtual-function-count: 7
    eno4:
      dhcp4: false
      virtual-function-count: 7

set up vfio-pci (put script somewhere and run on boot) /usr/local/lib/vfio-net-setup

#!/bin/bash
## change to your virtual devices' id from lspci
vdevid=8086:1520

echo "Setting up vfio-passthrough mode for $vdevid"
for dev in $(lspci -d $vdevid -n | cut -d' ' -f 1); do
  file="/sys/bus/pci/devices/0000:${dev}/driver/unbind"
  if [[ -f "$file" ]]; then
    echo "unbinding $dev"
    echo 0000:$dev > "$file"
  fi
done

echo "enabling vfio-pci"
echo ${vdevid/:/ } |> /sys/bus/pci/drivers/vfio-pci/new_id

make it run when libvirtd starts `mkdir /etc/systemd/system/libvirtd.service.d; vi /etc/systemd/system/libvirtd.service.d/vfio.conf"

[Service]
ExecStartPre=/usr/local/lib/vfio/vfio-net-setup

set up passthrough (duplicate for each network or add additional pfs)

<network>
  <name>port1</name>
  <forward mode='hostdev' managed='yes'>
    <pf dev='eno1'/>
  </forward>
</network>

Note: I wasn't able to get VLAN passthrough working, but I don't know whether it is a misconfiguration on the KVM side or a limitation with vfio-passthrough. Rather than figure it out I just set one of my switch ports to use the needed VLAN for untagged traffic and hooked it up to an unused port on the network card. Let me know if you figure it out.

vfio-net-setup

#!/bin/bash
## change to your virtual devices' id from lspci
vdevid=8086:1520

echo "Setting up vfio-passthrough mode for $vdevid"
for dev in $(lspci -d $vdevid -n | cut -d' ' -f 1); do
  file="/sys/bus/pci/devices/0000:${dev}/driver/unbind"
  if [[ -f "$file" ]]; then
    echo "unbinding $dev"
    echo 0000:$dev > "$file"
  fi
done

echo "enabling vfio-pci"
echo ${vdevid/:/ } |> /sys/bus/pci/drivers/vfio-pci/new_id