Machine Build: Char

Char.md

BIOS Update

My Gigabyte x570 Ultra shipped with Bios F5, which had no support for separate IOMMU groups.

I upgraded to F11, and set:

Tweaker > Advanced CPU Settings > SVM Mode > Enable
Settings > Miscellaneous > IOMMU > Enable
Settings > AMD CBS > ACS Enable > Enable
Settings > AMD CBS > Enable AER Cap > Enable

Fedora install

$ sudo vi /etc/dnf/dnf.conf
fastestmirror=true
max_parallel_downloads=10
$ sudo dnf update
$ sudo dnf add gparted vim

Add the software described here: https://qubitrenegade.com/virtualization/kvm/vfio/2019/07/17/VFIO-Fedora-Notes.html

$ lspci | grep VGA
0b:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
0c:00.0 VGA compatible controller: NVIDIA Corporation GF110 [GeForce GTX 580] (rev a1)
$ for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort
/sys/kernel/iommu_groups/30/devices/0000:0b:00.0
/sys/kernel/iommu_groups/30/devices/0000:0b:00.1
/sys/kernel/iommu_groups/31/devices/0000:0c:00.0
/sys/kernel/iommu_groups/31/devices/0000:0c:00.1
$ lsusb
Bus 001 Device 013: ID 04d9:1818 Holtek Semiconductor, Inc. Keyboard [Diatec Filco Majestouch 2]
Bus 001 Device 012: ID 046d:c00e Logitech, Inc. M-BJ58/M-BJ69 Optical Wheel Mouse

Swapped the GPUs to put the 580 first:

$ ./iommu.sh | grep -C 2 VGA
    00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 30: 
    0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF110 [GeForce GTX 580] [10de:1080] (rev a1)
    0b:00.1 Audio device [0403]: NVIDIA Corporation GF110 High Definition Audio Controller [10de:0e09] (rev a1)
IOMMU Group 31: 
    0c:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
    0c:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 32: 

https://arseniyshestakov.com/2016/03/31/how-to-pass-gpu-to-vm-and-back-without-x-restart/