Neverywhere/prevent-host-driver-in-pci-pass.md

## prevent-host-driver-in-pci-pass.md

      
    Raw
  

              prevent-host-driver-in-pci-pass.md
            
          
    Scenario

You're running a KVM-based virtualization. You want to do PCI/PCIe passthrough of some device. You don't want it to attach to the host OS at all.
Your device looks like this:
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

Problem

Usually the solutions are simple:
a. If you have only one device listing some module in Kernel modules under lspci -v (e.g. nvidiafb) you can add it to /etc/modprobe.d/some-file.conf as blacklist nvidiafb
b. If you have multiple and they're normal devices you just add options vfio-pci ids=8086:1c02 to some file in/etc/modprobe.d/ (make sure to use the vendor id found by running lspci -n -s 00:1f.2, and not the PCI location 00:1f.2)
However, these will not work if your device is handled by something loaded very early, like a driver for your second SATA controller.
a. You cannot blacklist ahci (like in example here) because you will prevent all controllers from working (=no boot volume)
b. You cannot use modprobe.d to set options because vfio-pci loads much too late.
Solution

There are two prerequisites:

vfio-pci must be available before rootfs is attached
vfio-pci must load before ahci does

1. The simple part:

add vfio-pci to /etc/initramfs-tools/modules
update initramfs: update-initramfs -u -k $(uname -r)
This will place the module in the initramfs disk (in /etc/conf/modules)

2. The only slightly harder part:
An entry in /etc/initramfs-tools/modules will load vfio-pci before the rootfs is mounted
However, /etc/conf/modules from ramdisk is loaded after some scripts (see /init in ramdisk)
These scripts (scripts/init-top/) load some drivers... and udev... and udev loads ahci

create /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci, and enter
#!/bin/sh
modprobe vfio-pci ids=8086:1c02

chmod +x /usr/share/initramfs-tools/scripts/init-top/load_vfio-pci
edit /usr/share/initramfs-tools/scripts/init-top/udev and change PREREQS="" to PREREQS="load_vfio-pci"
update initramfs: update-initramfs -u -k $(uname -r)
note: this will not work if placed in the "standard place" (/etc/initramfs-tools/scripts...) as dependencies are not cross-directory and /usr/share comes first

Verify

Without the mod:
# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: ahci
	Kernel modules: ahci

With the mod:
# lspci -knn
...
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
	Subsystem: Hewlett-Packard Company 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [103c:330d]
	Kernel driver in use: vfio-pci
	Kernel modules: ahci