Skip to content

Instantly share code, notes, and snippets.

@MatthiasLohr
Last active April 2, 2024 18:15
Show Gist options
  • Star 9 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save MatthiasLohr/8111bbdea62061f4cc3e01fe0bc8d071 to your computer and use it in GitHub Desktop.
Save MatthiasLohr/8111bbdea62061f4cc3e01fe0bc8d071 to your computer and use it in GitHub Desktop.
Beelink GTR5 + Proxmox + Windows Guest VM

Beelink GTR5 + Proxmox VE + Windows Guest using the Vega8 Graphics Card

Prerequisites

Hardware

  • Current BIOS version: 5.19
  • 1TB Kingston NVMe SSD (shipped with the device, used for system)
  • 1TB Samsung 860 EVO 2.5" SATA SSD (bought additionally, for Windows 10)
  • Installation images for Proxmox VE, Windows 10 (e.g. using Ventoy)

Set Up

BIOS Configuration

  • Advanced -> AMD CBS -> NBIO -> IOMMU: Enabled
  • Advanced -> AMD CBS -> NBIO -> GFX Configuration -> iGPU Configuration: UMA_SPECIFIED
  • Advanved -> AMD CBS -> NBIO -> GFX Configuration -> UMA Frame buffer Size: 8GB

Proxmox VE

Install Proxmox

The setup was done with Proxmox VE 7.1 (iso release 2). I followed the installation wizard without any special settings. As target harddisk, I chose /dev/nvme0n1 to use the (faster) NVMe SSD over the SATA SSD.

During my installation, it took a long time with status "create LVs". First, I tought setup is stuck here, but after a short Google search, I found that this happens from time to time. Unfortunately, I could not identify the actual reason for this, so I just decided to wait (approx. 15 minutes, then setup was progressing).

Since I'm using Proxmox VE privately and just for myself, I do not have an enterprise subscription. Therefore, I exchanged the APT package repository with the no-subscription repository (check https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_no_subscription_repo for more details). After the package lists are updated, I updated the system with apt update and apt dist-upgrade. To complete the update and load everything fresh, a reboot can be a good idea at this point.

Prepare PCI Passthrough

Enable/configure IOMMU

GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt amd_iommu=on video=efifb:off" update-grub check dmesg | grep -e DMAR -e IOMMU There should be a line that looks like "DMAR: IOMMU enabled". If there is no output, something is wrong.

[    0.364348] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.364837] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.400105] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    3.241752] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
Disable original GPU drivers

echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf update-initramfs -u

Load vfio Kernel Modules

/etc/modules:

vfio vfio_iommu_type1 vfio_pci vfio_virqfd

update-initramfs -u

Assign GPU to vfio

Use lspci or lspci -v to identify your GPU (here, it is 05:00). Use lspci -n -s 05:00 to get vendor IDs Here:

05:00.0 0300: 1002:1638 (rev c4)
05:00.1 0403: 1002:1637
05:00.2 1080: 1022:15df
05:00.3 0c03: 1022:1639
05:00.4 0c03: 1022:1639
05:00.5 0480: 1022:15e2 (rev 01)
05:00.6 0403: 1022:15e3

echo "options vfio-pci ids=1002:1638,1002:1637,1022:15df,1022:1639,1022:15e2,1022:15e3 disable_vga=1" > /etc/modprobe.d/vfio.conf

Windows 10 VM

Download Windows 10 ISO to /var/lib/vz/template/iso/.

Create a new VM and change the following options:

  • General
    • Start at boot: no (do not start the VM directly after creation!)
  • OS
    • ISO Image: The downloaded Windows Image
    • Type: Microsoft Windows
    • Version: 10/2016/2019
  • System
    • Graphics Card: Leave unchanged for now
    • SCSI Controller: VirtIO SCSI
    • Machine: q35
    • BIOS: OVMF
  • Disks
    • Keep unchanged for now
  • CPU
    • Cores: As many as you want to pass to your VM
  • Memory
    • Memory: As much as you want to pass to your VM
  • Network
    • Model: VirtIO (paravirtualized)

Now, let's add the SATA SSD:

ls /dev/disk/by-id/
qm set <VMID> -sata0 /dev/disk/by-id/ata-Samsung_SSD_860_EVO_1TB_S4X6NE0M706711V

In the GUI:

  • Hardware
    • Remove hard disk created on disk creation (we only want to have the SATA SSD)
    • Add another CD/DVD drive with the virtio driver ISO
  • Options
    • Boot Order: Ajust Boot Order

References

PCI Passthrough

Physical Disk Passthrough

@kchiem
Copy link

kchiem commented Apr 20, 2023

Does your GPU passthrough actually work?

I've followed the same steps, but my dmesg | grep -e DMAR -e IOMMU output is missing the last line: AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>. I am however on Proxmox 7.4 and the opt-in 6.2 kernel.

Your VM setup also skips the GPU passthrough step (Graphics Card: Leave unchanged for now). When I try to pass 05:00.0 to the VM, the entire system hangs on VM startup.

@MatthiasLohr
Copy link
Author

Unfortunately, no, no success for now. To be honest: I also stopped, since I ran out of ideas and other threads regarding this (like cosminmocan/vfio-single-amdgpu-passthrough#3) also stopped discussing this issue... If you find a solution, please let me know!

@kchiem
Copy link

kchiem commented Apr 20, 2023

Thanks for confirming. I haven't had any success either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment