Skip to content

Instantly share code, notes, and snippets.

@techhazard
Last active May 24, 2024 16:51
Show Gist options
  • Save techhazard/1be07805081a4d7a51c527e452b87b26 to your computer and use it in GitHub Desktop.
Save techhazard/1be07805081a4d7a51c527e452b87b26 to your computer and use it in GitHub Desktop.
NixOS: PCI Passthrough

PCI Passthrough

Warning: unfinished (but successfull!)

I did PCI passthrough on Archlinux and Debian with the old PCI-stub method (this was pre-4.0 era). And later I did PCI passthrough on the 4.1+ kernels on Arch and Ubuntu (16.10 I think?).

This is my attempt at doing the same on Nixos.

Requirements

  • Supported hardware ✅
  • Linux kernel 4.1+ ✅
  • PCI IDs of the guest GPU: 1002:67b1, 1002:aac8 ✅
  • Patience to figure this out ❓

As you may have noticed, I use the Arch Wiki page about PCI-passthrough as a guide

Current status:

I added the intel_iommu=on to my kernel parameters. (see the nixfile below) I then ran a slightly modified version of the "Ensuring that the groups are valid": ensure_iommu_groups_are_valid.sh (below)

Output:

[redacted]
IOMMU Group 1 	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 	01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290/390] [1002:67b1] (rev 80)
IOMMU Group 1 	01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
[redacted]

My GPU is apparently in the same group as the PCI bridge, but that's no problem since it's the only thing in the group. If you do have the same situation, don't forget to look at the Archwiki: Plugging_your_guest_GPU_in_an_unisolated_CPU-based_PCIe_slot

My PCI IDs are: 1002:67b1 (video) and 1002:aac8 (audio)

I added vfio vfio_iommu_type1 vfio_pci vfio_virqfd to my kernelModules and added options vfio-pci ids=1002:67b1,1002:aac8 as extraModprobeConfig (see nixfile)

After doing journalctl -b | grep -i -e iommu -e dmar I knew that IOMMU was enabled correctly

I added hardcoded paths to the current OVMF store in the qemu config, but it works now! I used this to find it:

ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$/

Then I created a VM to test it (see archwiki instructions, did I mention I use the archwiki?), and after install I added the PCI device.

And it's working!! (but not thouroughly tested)

My previous install (with ubuntu as host) used the disk storage backend, but libvirt on NixOS is currently compiled without support.

TODO

#!/bin/bash
# This is a slightly modified version of the script found at
# https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid
# It adds a tab to make the output prettier and sorts the output per-group
shopt -s nullglob;
for d in /sys/kernel/iommu_groups/*/devices/*;
do
n="${d#*/iommu_groups/*}";
n="${n%%/*}";
printf 'IOMMU Group %s \t' "$n";
lspci -nns "${d##*/}";
done | sort -h -k 3
# put this file in /etc/nixos/
# change the settings tagged with "CHANGE:"
# and add
# ./pci-passthrough.nix
# to /etc/nixos/configuration.nix in `imports`
{config, pkgs, ... }:
{
# CHANGE: intel_iommu enables iommu for intel CPUs with VT-d
# use amd_iommu if you have an AMD CPU with AMD-Vi
boot.kernelParams = [ "intel_iommu=on" ];
# These modules are required for PCI passthrough, and must come before early modesetting stuff
boot.kernelModules = [ "vfio" "vfio_iommu_type1" "vfio_pci" "vfio_virqfd" ];
# CHANGE: Don't forget to put your own PCI IDs here
boot.extraModprobeConfig ="options vfio-pci ids=1002:67b1,1002:aac8";
environment.systemPackages = with pkgs; [
virtmanager
qemu
OVMF
];
virtualisation.libvirtd.enable = true;
virtualisation.libvirtd.enableKVM = true;
# CHANGE: add your own user here
users.groups.libvirtd.members = [ "root" "your username"];
# CHANGE: use
# ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$//'
# to find your nix store paths
virtualisation.libvirtd.qemuVerbatimConfig = ''
nvram = [
"/nix/store/_SOME_HASH_-OVMF-2014-12-10/FV/OVMF.fd:/nix/store/_SOME_HASH_-OVMF-2014-12-10/FV/OVMF_VARS.fd"
]
'';
}
@alecmev
Copy link

alecmev commented May 7, 2017

Thanks for the guide! A couple of quick notes:

{
  virtualisation.libvirtd.qemuVerbatimConfig = ''
    nvram = [ "${pkgs.OVMF}/FV/OVMF.fd:${pkgs.OVMF}/FV/OVMF_VARS.fd" ]
  '';
}

Also, libvirtd.enableKVM is redundant (it's enabled by default, when libvirtd.enable is true).

@concatime
Copy link

concatime commented Jan 15, 2018

Why am I getting Could not detect a default hypervisor inside virt-manager?
Config.: https://github.com/concatime/dotfiles/blob/master/nixos/laptop.nix

@WhittlesJr
Copy link

@concatime, I think I get the same thing. I always just connect manually through File->Add Connection (keeping default options). Even when I select "Autoconnect," I still have to do it manually each time.

@techhazard, take a look at my fork if you get a chance. I refactored this into a configurable module, among other things. Works like a charm. So all I have to add to my configuration.nix is:

  pciPassthrough = {
    enable = true;
    pciIDs = "10de:13c2,10de:0fbb,1095:3114";
    libvirtUsers = [ "alexj" ];
  };

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment