Skip to content

Instantly share code, notes, and snippets.

@ArseniyShestakov

ArseniyShestakov/gpu-hotplug.md

Last active May 3, 2024 05:54
Show Gist options
  • Save ArseniyShestakov/dc152d080c65ebaa6781 to your computer and use it in GitHub Desktop.
Save ArseniyShestakov/dc152d080c65ebaa6781 to your computer and use it in GitHub Desktop.
Download ZIP
Current QEMU dual-VM configuration. Thanks to everyone who helped me!
Raw
gpu-hotplug.md

Check it's working

DRI_PRIME=1 glxinfo | grep OpenGL

Unbind AMD GPU from Radeon driver

echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id

Use it in QEMU

qemu-system-x86_64
-name "win81_amd"

-bios /usr/share/seabios/bios.bin
-enable-kvm

-usb
-device usb-kbd
-device usb-mouse

-vga none
-device qxl

-m 12G

-no-hpet
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic
-smp 4,sockets=1,cores=2,threads=2

-soundhw ac97

-M q35
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio

-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

-net none

-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0

-monitor telnet:127.0.0.1:1234,server,nowait

Now you need to remove PCI device

echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove

Rescan will attach it back to Radeon

echo 1 > /sys/bus/pci/rescan

Check it via PRIME once again

DRI_PRIME=1 glxinfo | grep OpenGL

Raw
qemu_shared
Introduction:
This is my configuration for VMs and I'm change it from time to time.
At moment I'm decide to use following configuration that work best for hardware I have:
- By default I have "i915.enable_hd_vgaarb=0" because most of time I don't need Nvidia GPU.
- When I only need VM with AMD HD6950 I boot with option unchanged to keep Intel DRI working on host
- Once I need to run both VMs simultaneously I just change it to "i915.enable_hd_vgaarb=1" at boot time.
- Have two sets of options for AMD VM because w/o Nvidia one I have more RAM and don't to use taskset
Hardware:
- ASRock Z87 Extreme6
- i7 4771 and 32GB RAM
- 60GB SSD (for /), 2x 250GB HDDs in RAID (for /home), 1TB HDD with NTFS partition for Windows content
- AMD HD 6950 1GB and Nvidia GTX 260
Software:
- Ubuntu 14.04
- Linux Kernel 3.19 with ACS and i915 VGAarbiter patches
- Preemption Model set to Low Latency Desktop
- Timer frequency set to 1000HZ
- QEMU 2.0.0~rc1+dfsg-0ubuntu3 from repositories
- In past I used QEMU 2.2.1 from ppa:syseleven-platform/virtualization, but then I decide to use older Nvidia driver instead.
Also with 2.2.1 I had weird disconnect issue in "Path of Exile" game while changing areas.
System configuration:
- /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.enable_hd_vgaarb=1 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream"
- /etc/modprobe.d/blacklist-gpu.conf:
blacklist radeon
blacklist nouveau
- /etc/qemu/bridge.conf:
allow br0
- /etc/sysctl.d/40-hugepage.conf:
vm.nr_hugepages = 8200
- /etc/fstab:
hugetlbfs /dev/hugepages hugetlbfs mode=1770,gid=78 0 0
Both VMs using same "backing files" that not modified in runtime:
- /win81-base.qcow2 [13GB only!], compressed
This QCOW2 image contain installed and configured Windows 8.1
- /dev/disk/by-uuid/0AD949C94647B394
It's NTFS partition on 1TB HDD with games and some software installed.
Those files are base for QCOW2 images that used by virtual machines:
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-amd.qcow2
qemu-img create -b /win81-base.qcow2 -f qcow2 /win81-writable-nvidia.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-amd.qcow2
qemu-img create -b /dev/disk/by-uuid/0AD949C94647B394 -f qcow2 /home/gameshdd-writable-nvidia.qcow2
So I basically store VM-specific changes to my system drive on SSD and changes for content stored on HDD RAID.
Kernel build:
apt-get install kernel-package
apt-get build-dep linux-image-$(uname -r)
apt-get build-dep linux-image-3.19.0-23-generic
apt-get source linux-image-$(uname -r)
apt-get source linux-image-3.19.0-23-generic
cd linux-*
# Now just moment to hate Ubuntu packaging, I do not use VBOX!
rm -rf ubuntu/vbox
# Comment out virtualbox crap
nano ubuntu/Makefile
# Apply patches (my patch files are above kernel directory)
# Personally I just used patch version published by someone on google drive
https://gist.github.com/ArseniyShestakov/768a426e79e2b901f9eb
patch -p1 < ../override_for_missing_acs_capabilities.patch
patch -p1 < ../i915_317.patch
# Copy your current kernel config
cp -vi /boot/config-`uname -r` .config
# Change some options for better performance
# Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop)
# (CONFIG_PREEMPT=y)
# Processor Type and Features > Timer Frequency > 1000 Hz
# (CONFIG_HZ_1000=y CONFIG_HZ=1000)
make menuconfig
# Now build kernel
fakeroot make-kpkg -j 8 --initrd kernel-image kernel-headers --append-to-version=-acs-igdarb-lowpremption-timer1000hz
# Install built packages
dpkg -i ../*acs-igdarb-lowpremption-timer1000*.deb
All useful information may be found there:
https://bbs.archlinux.org/viewtopic.php?id=162768
https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0
http://vfio.blogspot.com
My personal expirience of why I use some of options:
https://www.reddit.com/r/linux_gaming/comments/3dr62f/set_up_gpu_passthrough_without_using_the_cli/ct8gr34
https://www.reddit.com/r/linux_gaming/comments/2z2y7h/gpu_passthrough_or_how_to_play_any_game_at_near/cpfh76x
Raw
qemu_vm_amd
# This system use real network card (00:19.0) as I have two in my mobo.
#################################################
# Exclusive configuration when no other VM running (more RAM, all cores)
#################################################
qemu-system-x86_64 \
-name "win81_amd_exclusive" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 10G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 8,sockets=1,cores=4,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0
#################################################
# Shared configuration when Nvidia VM is used too:
#################################################
taskset -c 0,2,4,6 qemu-system-x86_64 \
-name "win81_amd" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 8G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-amd.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-amd.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net none \
\
-device ioh3420,bus=pcie.0,multifunction=on,port=2,chassis=2,id=root.2 \
-device vfio-pci,host=00:19.0,bus=root.2,addr=00.0
Raw
qemu_vm_nvidia
# Driver on guest is 335.23, it's last driver that no break KVM and Hyper-V
# Anyway this doesn't make much sense because I only run D3D9 games on that card and those not sensetive to old drivers
# Optional removal of existing data in Nvidia VM
# This VM used for testing so I drop all changes on ever reboot
rm /win81-writable-nvidia.qcow2
rm /home/gameshdd-writable-nvidia.qcow2
cp /win81-writable-nvidia.qcow2.backup /win81-writable-nvidia.qcow2
cp /home/gameshdd-writable-nvidia.qcow2.backup /home/gameshdd-writable-nvidia.qcow2
taskset -c 1,3,5,7 qemu-system-x86_64 \
-name "win81_nvidia" \
\
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
-device qxl \
\
-usb \
-device usb-mouse \
\
-m 7G \
-mem-path /dev/hugepages \
\
-no-hpet \
-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic \
-smp 4,sockets=1,cores=2,threads=2 \
\
-soundhw hda \
\
-M q35 \
\
-drive file=/win81-writable-nvidia.qcow2,id=disk1,format=qcow2,if=virtio \
-drive file=/home/gameshdd-writable-nvidia.qcow2,format=qcow2,if=virtio \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
\
-net nic,model=virtio \
-net bridge,br=br0 \
\
-serial stdio \
-nographic
# Nvidia reset. Boot to reset GPU between usages to avoid host freeze.
# Also make it set fans to minimum so it's not so loud when VM not used.
timeout 10 qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 512 \
-cpu host \
-rtc base=localtime \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin \
-boot menu=off,strict=on -vga none -nographic \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on
# Everything else only needed if i915 VGA arbitration not used
# I don't really use it because
##########################################
# kvm=off required to bypass Nvidia driver virtualization limitations
# Hyper-V hv_ options can't be used for same reason
# The way I got consistent VM startup:
- First boot up "win81_nvidia_reset" configuration with livecd that use Nouveau.
- Once monitor activated shutdown livecd or just kill QEMU.
Display will remain with frozen image on it (or shutdown black screen, but active).
- Now boot "win81_nvidia" configuration.
- In "Device Manager" disable and enable device.
Now monitor must lost connection and go idle / switch to different active port.
- Disable and enable device once again. It's should be working now.
qemu-system-x86_64 \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-enable-kvm \
\
-vga none \
\
-m 1G \
\
-cpu host \
-smp 1,sockets=1,cores=1,threads=1 \
\
-name "win81_nvidia_reset" \
-M q35 \
\
-device ioh3420,bus=pcie.0,addr=1c.0,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,x-vga=on \
\
-drive file=/opt/kubuntu-15.04-desktop-amd64.iso,id=isocd \
-device ide-cd,bus=ide.1,drive=isocd
Raw
vfio-setup.sh
#!/bin/sh
service lightdm stop
modprobe vfio-pci
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 6719" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
echo "1002 aa80" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:00:19.0" > /sys/bus/pci/devices/0000:00:19.0/driver/unbind
echo "0000:00:19.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "8086 153b" > /sys/bus/pci/drivers/vfio-pci/remove_id
echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind
echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo "10de 05e2" > /sys/bus/pci/drivers/vfio-pci/remove_id
service lightdm star
Raw
xorg.conf.d
/etc/X11/xorg.conf.d/10-intel.conf:
Section "Device"
Identifier "Intel Graphics"
Driver "intel"
Option "DRI" "3"
EndSection
/etc/X11/xorg.conf.d/20-radeon.conf:
Section "Device"
Identifier "Radeon"
Driver "radeon"
Option "DRI3" "1"
EndSection
@fld
Copy link

fld commented Dec 12, 2016

Hi,

What versions of Xorg/Mesa/Intel DDX are you using?

I'm on Debian jessie, running kernel v4.7.8-1~bpo8+1, using xorg-server v1.16.4-1 with backported xserver-xorg-video-intel v2.99.917+git20161105-1 and backported Mesa v12.0.4-2. I'm using Haswell graphics for desktop (with DRI3 enabled) and PRIME offloading games to Radeon HD7770. Works great, until I try unbinding the HD7770 from the radeon driver while Xorg is running... That's when the kernel just freezes dead :) Apparently you've managed to unbind your radeon while Xorg is running?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment