GPU Passthrough Fedora

readme.md

Simular Guides

• (My) Perfect Gaming VM Setup with GPU Passthrough – Virtualization Server Part 14
• Configuring GPU Passthrough with VFIO on Fedora 30 notes

Usefull Commands

• lspci -nnk list pci devices
• sudo grub2-mkconfig -o /etc/grub2-efi.cfg updates the grub
• sudo call-a-friend @ballas calls the best knowledge base, fuck wiki 🤣

Detach the GPU from the host

---

Retrieve the GPU id's

First you will have to find the GPU id's that you would like to passthrough, you can do this by running in the terminal:

# For AMD GPU's:
$ lspci -nnk | grep Navi

# For nvidia GPU's:
$ lspci -nnk | grep NVIDIA 

Example output, here we are interested in the VGA compatible controller (1002:73bf) and the Audio device (1002:ab28)

0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]

Blacklist id's and enable iommu

First you will have to modify the default grub file.

$ sudo gedit /etc/default/grub

We want to modify the GRUB_CMDLINE_LINUX

# for amd  gpu replace GPU_TYPE with `amdgpu`.
# for nvidia gpu replace GPU_TYPE with `nouveau`.
# if you have and intel replace `amd_iommu` with `intel_iommu`.
# `vfio-pci.ids` are equivalent to the id's found in the previous step.

# values to add
`amd_iommu=on`
`video=efifb:off` // disable gpu on boot
`kvm.ignore_msrs=1`
`rd.driver.pre=vfio-pci` // this preloads the vfio-pci

- GRUB_CMDLINE_LINUX="rhgb quiet"
+ GRUB_CMDLINE_LINUX="rhgb quiet rd.driver.blacklist=GPU_TYPE modprobe.blacklist=GPU_TYPE video=efifb:off amd_iommu=on rd.driver.pre=vfio-pci kvm.ignore_msrs=1 vfio-pci.ids=1002:73bf,1002:ab28"

Then we need to update grup

sudo grub2-mkconfig -o /etc/grub2-efi.cfg

We then need to edit the dracut config

$ sudo gedit /etc/dracut.conf.d/local.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "

$ sudo dracut -f --kver $(uname -r)

Rebooot!

Attach the GPU from the guest

---

Install Virtual Machine

$ sudo dnf groupinstall virtualization

Then you reboot 🦸

1. Open up Virtual Machine Manager
2. Create new virtual machine
3. Local install media (ISO image or CDROM) -> Select the ISO to windows10 or windows11
4. Assign Ram & CPU
5. Tick the Customize configuration before install

If you are running windows 11 you will need a secure boot and tpm module.

1. Under overview add the following config

Chipset: q35
Firmware: x86_64:/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd

2. Under CPUs, untick the copy host cpu configuration then for a example use the following config for a Ryzen 5900x

Model: host-passthrough
Topology
[X] Manually set CPU topology
Sockets: 1
Cores: 6
Threads: 2

3. Add a TPM module by clicking on Add Hardware > TPM

Model: TIS
Backend: Emulated device
Version: 2.0

4. Then passthrough the GPU and if needed the GPU Audio under the Add Hardware > PCI HOST Device

And then you can begin installation. Bobs your uncle 🤠

Hey, simple question, I have like two GPUs (one integrated from cpu):

00:02.0 Display controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 05)
DeviceName: Onboard - Video
Subsystem: ASUSTeK Computer Inc. Device 8694
Kernel driver in use: i915

01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. Device 1612
Kernel driver in use: nouveau
Kernel modules: nouveau

How does it work? Will my computer use the integrated GPU from the CPU if I disable the GPU (main, rtx 3080) from Booting?
Best regards
cian

Thank you so much for making this @paul-vd !

Hey, simple question, I have like two GPUs (one integrated from cpu):

00:02.0 Display controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 05)

DeviceName: Onboard - Video
Subsystem: ASUSTeK Computer Inc. Device 8694
Kernel driver in use: i915

01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1) Subsystem: ZOTAC International (MCO) Ltd. Device 1612 Kernel driver in use: nouveau Kernel modules: nouveau

How does it work? Will my computer use the integrated GPU from the CPU if I disable the GPU (main, rtx 3080) from Booting? Best regards cian

Yup, you will have to plug in display outputs to both your motherboards display out and your gpu's output. Also make sure that the gpu you're passing through exists in its own iommu group,

Essentially just run this script:

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

and see if the output puts your gpu you want to pass through in its own iommu group entirely (Should look something like this)

IOMMU Group 16:
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107 [GeForce GT 640] [10de:0fc1] (rev a1)
06:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
IOMMU Group 17:

(heres a snippet from the archwiki that goes through this in detail)

For anyone getting dracut-install: Failed to find module 'vfio_virqfd' error after the dracut config bit:

sudo gedit /etc/dracut.conf.d/local.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "

sudo dracut -f --kver $(uname -r)

Post Linux-kernel 6.2, vfio_virqfd has been consolidated with the vfio driver so you don't need to append it in the dracut config anymore. The fixed snippet would be:

sudo gedit /etc/dracut.conf.d/local.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci "

sudo dracut -f --kver $(uname -r)

- add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd "
+ add_drivers+=" vfio vfio_iommu_type1 vfio_pci "

To block Nvidia's proprietary driver modules use:
rd.driver.blacklist=nvidia,nvidiafb,nvidia-gpu modprobe.blacklist=nvidia,nvidiafb,nvidia-gpu

Source:
https://3os.org/infrastructure/proxmox/gpu-passthrough/igpu-passthrough-to-vm/#proxmox-configuration-for-igpu-full-passthrough

Hi! I followed this guide on my laptop with Ryzen 7 4700U and its GPU but, when I start the VM the screen begins black and the computer stucked. If I remove the PCI device all works correctly. How can I fix it? Is there anything I can share to make the problem clearer?