Skip to content

Instantly share code, notes, and snippets.

@tkolo
Last active March 14, 2024 06:26
Show Gist options
  • Save tkolo/b1f24130efec706bc66205badbca85cc to your computer and use it in GitHub Desktop.
Save tkolo/b1f24130efec706bc66205badbca85cc to your computer and use it in GitHub Desktop.
eGPU passthrough to VM for Pop!_OS/Ubuntu

Adapted from Arch wiki

Prerequisites

hardware

The extrnal monitor has to be attached to eGPU, not laptop. Otherwise this won't work.

libvirt & virt-manager

Install virt manager:

sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils

And add yourself to libvirt group, if not added automatically by apt:

sudo adduser `id -un` libvirtd

both commands from https://help.ubuntu.com/community/KVM/Installation

Next, install virt-manager:

sudo apt install virt-manager

iommu enaled

Add intel_iommu=on iommu=pt to your kernel parameters (cmdline). Not sure how that's done in plain Pop!_OS, consult support or documentation.

At this point, reboot your PC

Configuring VFIO driver

Save the following script as iommu.sh in your home directory:

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Mark it as executable (chmod +x iommu.sh) and execute it (./iommu.sh). It should list all IOMMU groups with all devices inside them. If there's no output, make sure that kernel parameters from previous step are set correctly. Now you need to find the group with your GPU. It'll look something like this:

IOMMU Group 20:
        05:01.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
        06:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
        07:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
        08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5700 / 5700 XT] [1002:731f] (rev c1)
        08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio [1002:ab38]

Now, create /etc/modprobe.d/vfio.conf file configuring the module with IDs from square brackets at the end of line. You can edit the file with command sudoedit /etc/modprobe.d/vifo.conf In my case, the configuration file resulting from the above output is this:

options vfio-pci ids=8086:15d3,1002:1478,1002:1479,1002:731f,1002:ab38

Change the ids part appropriately and save

Next, edit etc/initramfs-tools/modules (again, sudoedit) and add following four lines at the end:

vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd 

update initramfs with sudo update-initramfs -u and reboot

After rebooting, make sure that VFIO driver loaded and correctly grabbed appropriate devices with command dmesg | grep -i vfio (no sudo required). The output should be something like this:

[    2.592489] vfio_pci: add [8086:15d3[ffffffff:ffffffff]] class 0x000000/00000000
[    2.592492] vfio_pci: add [1002:1478[ffffffff:ffffffff]] class 0x000000/00000000
[    2.592495] vfio_pci: add [1002:1479[ffffffff:ffffffff]] class 0x000000/00000000
[    2.592498] vfio_pci: add [1002:731f[ffffffff:ffffffff]] class 0x000000/00000000
[    2.592501] vfio_pci: add [1002:ab38[ffffffff:ffffffff]] class 0x000000/00000000

At this point, we're ready to install windows

Installing windows

Widnows ISO

download latest windows 10 ISO from https://www.microsoft.com/en-US/software-download/windows10ISO

Virtio drivers ISO

download latest virtio drivers ISO from https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/ You'll need to select the folder with highest version number and download file virtio-win.iso. At the time of writing this guid it's virtio-win-0.1.173-4/virtio-win.iso.

We need a folder where we'll store the ISO files and VM's disk image. In this guide, it's assumed to be the same folder /home/VMs. If your configuration differs, adjust appropriately.

Installation

wizard

Launch virt-manager (should be available in application drawer) and begin adding new virtual machine (select File->New virtual machine or click plus button) In first step, select local installation from ISO

In second step, click browse, this will open new window. In the new window click plus button in lower right corner to add new localization to libvirt In the window that opens select type: dir, name it as you like and in path field type or select /home/VMs. Select newely created ppol from the left section and then windows ISO file from the main section, click Finish. If the system version was not detected automatically, uncheck the automatic detection checkbox and select windows 10

Third step should be self-explainatory. I recommend assigning no less than 4GB RAM to guest and all of the cores.

In fourth step select manual image selection, click Manage In the window that opens make sure the in the left section the pool pointing at /home/VMs is selected, then click plus button in the main section, above the table with available volumes. Name the volume as you like, make sure it's set to qcow2 and assign no less than 40GB of space (preferably much more).

In fifth and finall step make sure that the checkbox to adjust the configuration is enabled and click finish

configuration adjustment

THIS IS VERY IMPOTANT In overview section make sure that firmware is set to OVMF instead of BIOS, otherwise passthrough won't work. If you have multiple OVMF images, select the one with no suffixes (shortest name). If there's none, search for and install ovmf package (probably named simlpy ovmf but not sure)

In Processor/CPU section make sure that checkbox indicating configuration passthrough is enabled. As optimization expand topology section and check manual topology, then configure number of cores as it is listed in lscpu (most likely 1 socket, 4 cores and 2 threads).

In disk section expand advanced section and set driver to VirtIO

In network section also set driver to VirtIO

Add additional CD-ROM drive and point it at virtio-win.iso. Make sure that device type is CD-ROM and controller is SATA.

You can now click Beign installation in upper-right corner

IMPORTANT The machine might take a minute or two to launch, yeilding high CPU load during initialization, not sure what's that about. It works afterwards.

installation

During installation windows won't be able to find the harddrive, the CD-ROM with virtio-win will have the appropriate drivers.

Windows configuration

After installing, launch explorer, go to CD-ROM representing with virtio-win.iso and install all other drivers, there should be installer executable inside that streamlines the process. Next, go to your GPU vendor's site and download and install GPU drivers for the graphic card

After installing everything, shut windows down.

Adding PCI card to VM and removing SPICE channel

go back to virt-manager, select your VM and go to devices section (the "I" icon) Remove the

  • Both CDROMs (optional)
  • Spice display
  • Tablet
  • QXL GPU
  • Parallel port
  • Spice channel
  • VirtIO Serial controller
  • All USB redirects

Or otherwise, remove everything you can that isn't hard drive, network interface or sound card

Next, add new device (button in lower left corner), Select "Host's PCI device" and on the list select your GPU. If there are two entries, because i.e. it also appears as soundcard make sure to redirect both of them

Now, we have an issue of mouse & keyboard. If you have external USB mouse and keyboard I'd recommend passing them through to VM, so that windows has it's own mouse & keyboard while Linxu uses laptop's keyboard and touchpad (they're not redirectable anyway). To do that, add new device and select "Host's USB device", then select appropriate entries. If you do not have spare mouse and keyboard then you can follow this section from Arch wiki, though I haven't tested it myself.

Now you can boot the VM again, everything should work as intended execept there'll be no sound

Making sound work

Make sure that VM is powered down.

Sudoedit /etc/libvirt/qemu.conf file, set user to your user and group to your group. Also set nographics_allow_host_audio to 1, save and exit editor.

Next, sudoedit the file describing your virtualmachine that's located in /etc/libvirt/qemu directory. If your VM is named win10 then file will be named win10.xml. Change first line from

<domain type='kvm'>

to

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Before </domain> (last line) add:

<qemu:commandline>
  <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
  <qemu:env name='QEMU_PA_SAMPLES' value='8192'/>
  <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/>
  <qemu:env name='QEMU_PA_SERVER' value='unix:/run/user/1000/pulse/native'/>
</qemu:commandline>

1000 is assumed to be your user's ID, you can check if that's the case with id -u. If it's not ,adjust <qemu:env name='QEMU_PA_SERVER' value='unix:/run/user/1000/pulse/native'/> appropriately.

Now, restart libvirt with sudo systemctl restart libvirtd or simply reboot. After that, sound should work in VM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment