Skip to content

Instantly share code, notes, and snippets.

@sheerun
Forked from techhazard/Readme.md
Created November 3, 2023 14:45
Show Gist options
  • Save sheerun/4dc9f01b7e1348875a135b118521410e to your computer and use it in GitHub Desktop.
Save sheerun/4dc9f01b7e1348875a135b118521410e to your computer and use it in GitHub Desktop.
NixOS: PCI Passthrough

PCI Passthrough

Warning: unfinished (but successfull!)

I did PCI passthrough on Archlinux and Debian with the old PCI-stub method (this was pre-4.0 era). And later I did PCI passthrough on the 4.1+ kernels on Arch and Ubuntu (16.10 I think?).

This is my attempt at doing the same on Nixos.

Requirements

  • Supported hardware ✅
  • Linux kernel 4.1+ ✅
  • PCI IDs of the guest GPU: 1002:67b1, 1002:aac8 ✅
  • Patience to figure this out ❓

As you may have noticed, I use the Arch Wiki page about PCI-passthrough as a guide

Current status:

I added the intel_iommu=on to my kernel parameters. (see the nixfile below) I then ran a slightly modified version of the "Ensuring that the groups are valid": ensure_iommu_groups_are_valid.sh (below)

Output:

[redacted]
IOMMU Group 1 	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 	01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290/390] [1002:67b1] (rev 80)
IOMMU Group 1 	01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio [1002:aac8]
[redacted]

My GPU is apparently in the same group as the PCI bridge, but that's no problem since it's the only thing in the group. If you do have the same situation, don't forget to look at the Archwiki: Plugging_your_guest_GPU_in_an_unisolated_CPU-based_PCIe_slot

My PCI IDs are: 1002:67b1 (video) and 1002:aac8 (audio)

I added vfio vfio_iommu_type1 vfio_pci vfio_virqfd to my kernelModules and added options vfio-pci ids=1002:67b1,1002:aac8 as extraModprobeConfig (see nixfile)

After doing journalctl -b | grep -i -e iommu -e dmar I knew that IOMMU was enabled correctly

I added hardcoded paths to the current OVMF store in the qemu config, but it works now! I used this to find it:

ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$/

Then I created a VM to test it (see archwiki instructions, did I mention I use the archwiki?), and after install I added the PCI device.

And it's working!! (but not thouroughly tested)

My previous install (with ubuntu as host) used the disk storage backend, but libvirt on NixOS is currently compiled without support.

TODO

#!/bin/bash
# This is a slightly modified version of the script found at
# https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid
# It adds a tab to make the output prettier and sorts the output per-group
shopt -s nullglob;
for d in /sys/kernel/iommu_groups/*/devices/*;
do
n="${d#*/iommu_groups/*}";
n="${n%%/*}";
printf 'IOMMU Group %s \t' "$n";
lspci -nns "${d##*/}";
done | sort -h -k 3
# put this file in /etc/nixos/
# change the settings tagged with "CHANGE:"
# and add
# ./pci-passthrough.nix
# to /etc/nixos/configuration.nix in `imports`
{config, pkgs, ... }:
{
# CHANGE: intel_iommu enables iommu for intel CPUs with VT-d
# use amd_iommu if you have an AMD CPU with AMD-Vi
boot.kernelParams = [ "intel_iommu=on" ];
# These modules are required for PCI passthrough, and must come before early modesetting stuff
boot.kernelModules = [ "vfio" "vfio_iommu_type1" "vfio_pci" "vfio_virqfd" ];
# CHANGE: Don't forget to put your own PCI IDs here
boot.extraModprobeConfig ="options vfio-pci ids=1002:67b1,1002:aac8";
environment.systemPackages = with pkgs; [
virtmanager
qemu
OVMF
];
virtualisation.libvirtd.enable = true;
virtualisation.libvirtd.enableKVM = true;
# CHANGE: add your own user here
users.groups.libvirtd.members = [ "root" "your username"];
# CHANGE: use
# ls /nix/store/*OVMF*/FV/OVMF{,_VARS}.fd | tail -n2 | tr '\n' : | sed -e 's/:$//'
# to find your nix store paths
virtualisation.libvirtd.qemuVerbatimConfig = ''
nvram = [
"/nix/store/_SOME_HASH_-OVMF-2014-12-10/FV/OVMF.fd:/nix/store/_SOME_HASH_-OVMF-2014-12-10/FV/OVMF_VARS.fd"
]
'';
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment