Definitely prevent stubborn devices from being bound by the host driver in PCI passthrough scenario

prevent-host-driver-in-pci-pass.md

Scenario

You're running a KVM-based virtualization. You want to do PCI/PCIe passthrough of some device. You don't want it to attach to the host OS at all.

Your device looks like that:

00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

Problem

Usually the solutions are simple:

1. If you have only one device listing some module in Kernel modules (e.g. nvidiafb) you can add it to /etc/modprobe.d/some-file.conf as blacklist nvidiafb
2. If you have multiple and they're normal devices you just add options vfio-pci ids=8086:1c02 to some file in/etc/modprobe.d/ (make sure to use the id in [...] and not pci location 00:1f.2)

However, these will not work if your device is handled by something loaded very very VERY early... like a driver for your second SATA controller.

1. You cannot blacklist ahci (like in example here) because you will prevent all controllers from working (=no boot volume)
2. You cannot use modprobe.d to set options because vfio-pci loads waaaaay too late.

Solution

There are two prerequisites:

1. vfio-pci must be availbale before rootfs is attached
2. vfio-pci must load before ahci loads

The first is simple:

• add vfio-pci to /etc/initramfs-tools/modules
• update initramfs: update-initramfs -u -k $(uname -r)
• Proxmox on UEFI: if you're using Proxmox 7 booted using UEFI mode you also need to run proxmox-boot-tool refresh
• it will place the module in initramfs disk (in /etc/conf/modules)

The second is more complicated:

• entry in /etc/initramfs-tools/modules will load vfio-pci before the rootfs is mounted
• however, /etc/conf/modules from ramdisk is loaded after some scripts (see /init in ramdisk)
• these scripts (scripts/init-top/) load some drivers... and udev... and udev loads ahci
• solution:
  • create /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci with

    #!/bin/sh
    modprobe vfio-pci ids=8086:1c02

  • chmod +x /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci
  • edit /usr/share/initramfs-tools/scripts/init-top/udev and change PREREQS="" to PREREQS="load_vfio-pci"
• update initramfs: update-initramfs -u -k $(uname -r)
• Proxmox on UEFI: if you're using Proxmox 7 booted using UEFI mode you also need to run proxmox-boot-tool refresh
• note: this will not work if placed in "standard place" (/etc/initramfs-tools/scripts...) as dependencies are not cross-directory and /usr/share comes first

Verify

Without the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

With the mod:

# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: vfio-pci
	Kernel modules: ahci