orther/Readme.md

## Readme.md

      
    Raw
  

              Readme.md
            
          
    Things I changed in this fork

Modified WhittlesJr's forked version to prevent some devices from loading its drivers.
In my case, NVIDIA's USB Host controller is in the same IOMMU group, and specifying vfio-pci option didn't prevent the device from loading xhci_usb driver.
IOMMU Group 1   00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1   01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106M [GeForce RTX 2060 Mobile] [10de:1f11] (rev a1)
IOMMU Group 1   01:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
IOMMU Group 1   01:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
IOMMU Group 1   01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:1adb] (rev a1)

https://alexbakker.me/post/nixos-pci-passthrough-qemu-vfio.html
I then found a blog article describing how to override drivers for some devices. Instead of passing vfio-pci to driver_override (didn't seem to work for my case), I passed none to prevent USB controller from loading xhci_usb driver.
As you can see in the lspci output below, no drives will be loaded for the device with this configuration.
01:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
        Subsystem: Dell Device [1028:0000]
        Kernel modules: xhci_pci

PCI Passthrough (Original Readme.md)

Warning: unfinished (but successfull!)

I did PCI passthrough on Archlinux and Debian with the old PCI-stub method (this was pre-4.0 era).
And later I did PCI passthrough on the 4.1+ kernels on Arch and Ubuntu (16.10 I think?).
This is my attempt at doing the same on Nixos.
Requirements


Supported hardware ✅
Linux kernel 4.1+  ✅
PCI IDs of the guest GPU: 1002:67b1, 1002:aac8 ✅
Patience to figure this out ❓

As you may have noticed, I use the Arch Wiki page about PCI-passthrough as a guide
Current status:

I added the intel_iommu=on to my kernel parameters. (see the nixfile below)
I then ran a slightly modified version of the "Ensuring that the groups are valid": ensure_iommu_groups_are_valid.sh (below)
Output:
[redacted]
IOMMU Group 1 	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 	01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290/390] [1002:67b1] (rev 80)
IOMMU Group 1 	01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
[redacted]

My GPU is apparently in the same group as the PCI bridge, but that's no problem since it's the only thing in the group.
If you do have the same situation, don't forget to look at the Archwiki: Plugging_your_guest_GPU_in_an_unisolated_CPU-based_PCIe_slot
My PCI IDs are: 1002:67b1 (video) and 1002:aac8 (audio)
I added vfio vfio_iommu_type1 vfio_pci vfio_virqfd to my kernelModules and added options vfio-pci ids=1002:67b1,1002:aac8 as extraModprobeConfig (see nixfile)
After doing journalctl -b | grep -i -e iommu -e dmar I knew that IOMMU was enabled correctly
I added hardcoded paths to the current OVMF store in the qemu config, but it works now! I used this to find it:
ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$/

Then I created a VM to test it (see archwiki instructions, did I mention I use the archwiki?), and after install I added the PCI device.
And it's working!! (but not thouroughly tested)
My previous install (with ubuntu as host) used the disk storage backend, but libvirt on NixOS is currently compiled without support.
TODO


 Make OVMF paths more dynamic (perhaps create a package, or change the libvirt package and add an optional dependency?)
 Test patched libvirt version that is compiled with --with-storage-disk option and publish patch if succesful
 Clean up this guide
 Mention archwiki more


## ensuring_iommu_groups_are_valid.sh
#!/bin/bash
# This is a slightly modified version of the script found at
#    https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid
# It adds a tab to make the output prettier and sorts the output per-group
shopt -s nullglob;
for d in /sys/kernel/iommu_groups/*/devices/*;
do
        n="${d#*/iommu_groups/*}";
        n="${n%%/*}";
        printf 'IOMMU Group %s \t' "$n";
        lspci -nns "${d##*/}";
done | sort -h -k 3

## pci-passthrough.nix
# put this file in /etc/nixos/
# and add
#   ./pci-passthrough.nix
# to /etc/nixos/configuration.nix in `imports`

{config, pkgs, lib, ... }:

with lib;
let
  cfg = config.pciPassthrough;
in
{
  ###### interface
  options.pciPassthrough = {
    enable = mkEnableOption "PCI Passthrough";

    cpuType = mkOption {
      description = "One of `intel` or `amd`";
      default = "intel";
      type = types.str;
    };

    bcfIDs = mkOption {
      description = "list of BCF IDs to disable driver loading";
      type = types.listOf types.str;
      default = [];
    };

    pciIDs = mkOption {
      description = "list of PCI IDs to pass-through";
      type = types.listOf types.str;
    };

    libvirtUsers = mkOption {
      description = "Extra users to add to libvirtd (root is already included)";
      type = types.listOf types.str;
      default = [];
    };
  };

  ###### implementation
  config = (mkIf cfg.enable {

    boot.kernelParams = [ "${cfg.cpuType}_iommu=on" "iommu=pt" ];

    # These modules are required for PCI passthrough, and must come before early modesetting stuff
    boot.kernelModules = [ "vfio" "vfio_iommu_type1" "vfio_pci" "vfio_virqfd" "xhci_pci" ];
    boot.extraModprobeConfig = "options vfio-pci ids=${concatStringsSep "," cfg.pciIDs}";
    boot.initrd.preDeviceCommands = ''
      DEVS='${concatStringsSep " " cfg.bcfIDs}'
      for DEV in $DEVS; do
          echo 'none' > /sys/bus/pci/devices/0000:"$DEV"/driver_override
      done
    '';

    environment.systemPackages = with pkgs; [
      virtmanager
      qemu
      pciutils
    ];

    virtualisation.libvirtd.enable = true;
    virtualisation.libvirtd.qemuPackage = pkgs.qemu_kvm;

    users.groups.libvirtd.members = [ "root" ] ++ cfg.libvirtUsers;

    # FIXME: support users other than root
    systemd.tmpfiles.rules = ["f /dev/shm/looking-glass	0600 root kvm -"];
  });
}
	#!/bin/bash
	# This is a slightly modified version of the script found at
	# https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid
	# It adds a tab to make the output prettier and sorts the output per-group
	shopt -s nullglob;
	for d in /sys/kernel/iommu_groups//devices/;
	do
	n="${d#/iommu_groups/}";
	n="${n%%/*}";
	printf 'IOMMU Group %s \t' "$n";
	lspci -nns "${d##*/}";
	done \| sort -h -k 3
	# put this file in /etc/nixos/
	# and add
	# ./pci-passthrough.nix
	# to /etc/nixos/configuration.nix in `imports`

	{config, pkgs, lib, ... }:

	with lib;
	let
	cfg = config.pciPassthrough;
	in
	{
	###### interface
	options.pciPassthrough = {
	enable = mkEnableOption "PCI Passthrough";

	cpuType = mkOption {
	description = "One of `intel` or `amd`";
	default = "intel";
	type = types.str;
	};

	bcfIDs = mkOption {
	description = "list of BCF IDs to disable driver loading";
	type = types.listOf types.str;
	default = [];
	};

	pciIDs = mkOption {
	description = "list of PCI IDs to pass-through";
	type = types.listOf types.str;
	};

	libvirtUsers = mkOption {
	description = "Extra users to add to libvirtd (root is already included)";
	type = types.listOf types.str;
	default = [];
	};
	};

	###### implementation
	config = (mkIf cfg.enable {

	boot.kernelParams = [ "${cfg.cpuType}_iommu=on" "iommu=pt" ];

	# These modules are required for PCI passthrough, and must come before early modesetting stuff
	boot.kernelModules = [ "vfio" "vfio_iommu_type1" "vfio_pci" "vfio_virqfd" "xhci_pci" ];
	boot.extraModprobeConfig = "options vfio-pci ids=${concatStringsSep "," cfg.pciIDs}";
	boot.initrd.preDeviceCommands = ''
	DEVS='${concatStringsSep " " cfg.bcfIDs}'
	for DEV in $DEVS; do
	echo 'none' > /sys/bus/pci/devices/0000:"$DEV"/driver_override
	done
	'';

	environment.systemPackages = with pkgs; [
	virtmanager
	qemu
	pciutils
	];

	virtualisation.libvirtd.enable = true;
	virtualisation.libvirtd.qemuPackage = pkgs.qemu_kvm;

	users.groups.libvirtd.members = [ "root" ] ++ cfg.libvirtUsers;

	# FIXME: support users other than root
	systemd.tmpfiles.rules = ["f /dev/shm/looking-glass 0600 root kvm -"];
	});
	}