NixOS: PCI Passthrough

Readme.md

PCI Passthrough

Warning: unfinished (but successfull!)

I did PCI passthrough on Archlinux and Debian with the old PCI-stub method (this was pre-4.0 era). And later I did PCI passthrough on the 4.1+ kernels on Arch and Ubuntu (16.10 I think?).

This is my attempt at doing the same on Nixos.

Requirements

• Supported hardware ✅
• Linux kernel 4.1+ ✅
• PCI IDs of the guest GPU: 1002:67b1, 1002:aac8 ✅
• Patience to figure this out ❓

As you may have noticed, I use the Arch Wiki page about PCI-passthrough as a guide

Current status:

I added the intel_iommu=on to my kernel parameters. (see the nixfile below) I then ran a slightly modified version of the "Ensuring that the groups are valid": ensure_iommu_groups_are_valid.sh (below)

Output:

[redacted]
IOMMU Group 1 	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 	01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290/390] [1002:67b1] (rev 80)
IOMMU Group 1 	01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
[redacted]

My GPU is apparently in the same group as the PCI bridge, but that's no problem since it's the only thing in the group. If you do have the same situation, don't forget to look at the Archwiki: Plugging_your_guest_GPU_in_an_unisolated_CPU-based_PCIe_slot

My PCI IDs are: 1002:67b1 (video) and 1002:aac8 (audio)

I added vfio vfio_iommu_type1 vfio_pci vfio_virqfd to my kernelModules and added options vfio-pci ids=1002:67b1,1002:aac8 as extraModprobeConfig (see nixfile)

After doing journalctl -b | grep -i -e iommu -e dmar I knew that IOMMU was enabled correctly

I added hardcoded paths to the current OVMF store in the qemu config, but it works now! I used this to find it:

ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$/

Then I created a VM to test it (see archwiki instructions, did I mention I use the archwiki?), and after install I added the PCI device.

And it's working!! (but not thouroughly tested)

My previous install (with ubuntu as host) used the disk storage backend, but libvirt on NixOS is currently compiled without support.

TODO

• Make OVMF paths more dynamic (perhaps create a package, or change the libvirt package and add an optional dependency?)
• Test patched libvirt version that is compiled with --with-storage-disk option and publish patch if succesful
• Clean up this guide
• Mention archwiki more

ensuring_iommu_groups_are_valid.sh

#!/bin/bash
# This is a slightly modified version of the script found at 
#    https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid
# It adds a tab to make the output prettier and sorts the output per-group
shopt -s nullglob;
for d in /sys/kernel/iommu_groups/*/devices/*;
do
        n="${d#*/iommu_groups/*}";
        n="${n%%/*}";
        printf 'IOMMU Group %s \t' "$n";
        lspci -nns "${d##*/}";
done | sort -h -k 3

pci-passthrough.nix

# put this file in /etc/nixos/
# and add 
#   ./pci-passthrough.nix
# to /etc/nixos/configuration.nix in `imports`

{config, pkgs, lib, ... }:

with lib;
let
  cfg = config.pciPassthrough;
in
{
  ###### interface
  options.pciPassthrough = {
    enable = mkEnableOption "PCI Passthrough";

    cpuType = mkOption {
      description = "One of `intel` or `amd`";
      default = "intel";
      type = types.str;
    };

    pciIDs = mkOption {
      description = "Comma-separated list of PCI IDs to pass-through";
      type = types.str;
    };

    libvirtUsers = mkOption {
      description = "Extra users to add to libvirtd (root is already included)";
      type = types.listOf types.str;
      default = [];
    };
  };

  ###### implementation
  config = (mkIf cfg.enable {

    boot.kernelParams = [ "${cfg.cpuType}_iommu=on" ];

    # These modules are required for PCI passthrough, and must come before early modesetting stuff
    boot.kernelModules = [ "vfio" "vfio_iommu_type1" "vfio_pci" "vfio_virqfd" ];

    boot.extraModprobeConfig ="options vfio-pci ids=${cfg.pciIDs}";

    environment.systemPackages = with pkgs; [
      virtmanager
      qemu
      OVMF
      pciutils
    ];

    virtualisation.libvirtd.enable = true;
    virtualisation.libvirtd.qemuPackage = pkgs.qemu_kvm;

    users.groups.libvirtd.members = [ "root" ] ++ cfg.libvirtUsers;

    virtualisation.libvirtd.qemuVerbatimConfig = ''
      nvram = [
      "${pkgs.OVMF}/FV/OVMF.fd:${pkgs.OVMF}/FV/OVMF_VARS.fd"
      ]
    '';
  });

}

Thanks for sharing!

added these kernelParams in my case

      boot.kernelParams = [
        "${cfg.cpuType}_iommu=on"
        "kvm.ignore_msrs=1"
        "iommu=pt"
      ];

And these blacklisted kernel Modules

      # ===== My added conf =====
      boot.blacklistedKernelModules = [
        "nvidia"
        "nouveau"
      ];
      # ===== My added conf =====

Thanks for sharing 👍