kaapstorm/gpu_passthrough_on_trillian.rst

## gpu_passthrough_on_trillian.rst

      
    Raw
  

              gpu_passthrough_on_trillian.rst
            
          
    GPU Passthrough on trillian


Prepare the host


Configure primary GPU

Check that the integrated GPU is the primary GPU.

Configure IOMMU


If necessary, enable IOMMU in the BIOS.

Edit /etc/default/grub. Append intel_iommu=on to the value of GRUB_CMDLINE_LINUX_DEFAULT:
...
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"
...


Update grub:
$ sudo update-grub


Reboot

Identify the PCIe bus the GPU we're passing through is on:
$ lspci -nnk
...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750] [10de:1381] (rev a2)
    Subsystem: Gigabyte Technology Co., Ltd GM107 [GeForce GTX 750] [1458:362e]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
    Subsystem: Gigabyte Technology Co., Ltd Device [1458:362e]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
...


Check what group the graphics card for Windows is in:
$ find /sys/kernel/iommu_groups/ -type l


Check that only the graphics card for Windows is in the IOMMU group:
$ ls -lhA /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices/
total 0
... 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
... 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
... 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1


Configure pci-stub


Blacklist the GPU we're passing through to the VM so that the graphics driver can't grab it. We use the pci-stub module to claim the card before nvidia or nouveau can. Add "pci-stub" to /etc/initramfs-tools/modules:
$ echo "pci_stub" | sudo tee -a /etc/initramfs-tools/modules


(Credit: Weber K.) You can add options to kernel modules in /etc/initramfs-tools/modules, but we will add them in /lib/modprobe.d/pci-stub.conf. Pass pci-stub the IDs of the graphics controller and the audio device (as found with lspci -nnk above). Set pci-stub as a dependency for drm, otherwise the graphics driver will be loaded before the pci-stub driver. (Check dmesg to see when this happens.)
$ sudo vim /lib/modprobe.d/pci-stub.conf
options pci-stub ids=10de:1381,10de:0fbc
softdep drm pre: pci-stub


Update the existing initramfs image:
$ sudo update-initramfs -u


Reboot

Confirm that pci-stub claimed the devices:
$ lspci -nnk
...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750] [10de:1381] (rev a2)
    Subsystem: Gigabyte Technology Co., Ltd GM107 [GeForce GTX 750] [1458:362e]
    Kernel driver in use: pci-stub
    Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
    Subsystem: Gigabyte Technology Co., Ltd Device [1458:362e]
    Kernel driver in use: pci-stub
    Kernel modules: snd_hda_intel
...


Create script to bind to devices


Create a "vfio-bind" script to replace the pci-stub placeholder
driver with vfio-pci. Put it in /usr/local/bin/vfio-bind:
$ sudo vim /usr/local/bin/vfio-bind
#!/bin/bash

modprobe vfio-pci

for dev in "$@"
do
    vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
    device=$(cat /sys/bus/pci/devices/$dev/device)
    if [ -e /sys/bus/pci/devices/$dev/driver ]; then
        echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
    fi
    echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

$ sudo chmod +x /usr/local/bin/vfio-bind


Run the script with the PCIe bus IDs of the devices in the IOMMU group:
$ sudo vfio-bind 0000:01:00.0 0000:01:00.1


Confirm that the devices are now using the vfio-pci driver:
$ lspci -nnk
...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107 [GeForce GTX 750] [10de:1381] (rev a2)
    Subsystem: Gigabyte Technology Co., Ltd GM107 [GeForce GTX 750] [1458:362e]
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
    Subsystem: Gigabyte Technology Co., Ltd Device [1458:362e]
    Kernel driver in use: vfio-pci
    Kernel modules: snd_hda_intel
...


Create script for Windows VM


Install QEMU, KVM, and the OVMF UEFI BIOS:
$ sudo apt-get install qemu-kvm ovmf


You will need to run QEMU as root in order for it to use the vfio
device. You will need to give root permission to use your display
using a .Xauthority file. Here is a wrapper to do that. (Credit:
ton)
$ vim ~/bin/xsudo
#!/bin/sh

touch ~/.Xauthority
xauth generate :0 . trusted
sudo $@
rm ~/.Xauthority


Copy the OVMF variables image to support UEFI variables:
$ cp /usr/share/OVMF/OVMF_VARS.fd ovmf_vars.fd


You will need to dedicate a keyboard and mouse to Windows. Use
lsusb to find their vendor and product IDs:
$ lsusb
...
Bus 003 Device 004: ID 2516:0015 Cooler Master Co., Ltd. Keyboard
Bus 003 Device 007: ID 046d:c083 Logitech, Inc.
...


Create a script for your VM.

Here is the script. We will unpack it next.
$ vim windows
#!/bin/sh

vfio-bind 0000:01:00.0 0000:01:00.1
qemu-system-x86_64 \
    -enable-kvm \
    -monitor stdio \
    -name win10 \
    \
    -machine type=q35,accel=kvm \
    -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=NoFortyThree \
    -smp 3 \
    -m 8G \
    -usb \
    -vga none \
    \
    -device ich9-intel-hda \
    -device hda-micro,audiodev=hda \
    -audiodev pa,id=hda,server=unix:/run/user/$UID/pulse/native \
    \
    -drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
    -drive if=pflash,format=raw,file=ovmf_vars.fd \
    \
    -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=01:00.1 \
    \
    -device usb-host,vendorid=0x2516,productid=0x0015 \
    -device usb-host,vendorid=0x046d,productid=0xc083 \
    \
    -drive index=0,media=disk,file=$WINDOWS_IMG,format=raw \

$ chmod +x windows


We chose -machine type=q35 to use a PCIe bus.

We told the VM to hide the fact that we are using KVM with
kvm=off. This is to avoid a "bug" in NVIDIA's driver for
Windows.

We enabled Hyper-V enlightenments with
hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time.

We set hv_vendor_id=NoFortyThree to avoid the NVIDIA driver
throwing Error 43.

We chose Intel HD Audio for audio devices. We will need to run
the script as root. For root to use normal user 1001's pulseaudio
server, we need to copy their pulseaudio config to root's:
$ sudo mkdir -p /root/.config
$ sudo cp -a ~/.config/pulse /root/.config/
$ sudo chown -R root.root /root/.config/pulse


We are using drive interface if=pflash for the BIOS to support
UEFI variables.

GPU video and audio use the vfio-pci device.

The dedicated keyboard and mouse are set with the usb-host
device.


Set up the Windows VM

If you are reinstalling Windows:

Wipe all file system data:
$ sudo wipefs -a --force /dev/sdxX


Use qemu-img to write "MSEdge - Win10-disk001.vmdk" over the old drive:
$ sudo qemu-img dd -f vmdk -O raw bs=1M if="MSEdge - Win10-disk001.vmdk" of=/dev/sdxX


Skip to "Convert the VM to UEFI" below.


Create the Windows drive image


Download a Windows VM for VirtualBox from developer.microsoft.com.

If you can, use a 50GB native disk partition for Windows. (We will refer to it as /dev/sdxX.) Otherwise, use a "raw"-format file.

If you are using a partition, write the Windows VM drive image to the partition:
$ unzip MSEdge.Win10.VirtualBox.zip
$ tar -xvf "MSEdge - Win10.ova"
$ sudo qemu-img dd -f vmdk -O raw bs=1M if="MSEdge - Win10-disk001.vmdk" of=/dev/sdxX


If you are using a file, convert the drive image to raw format:
$ unzip MSEdge.Win10.VirtualBox.zip
$ tar -xvf "MSEdge - Win10.ova"
$ sudo qemu-img convert -f vmdk -O raw "MSEdge - Win10-disk001.vmdk" windows.img


Update the VM script.
$ vim windows
...
# /dev/sdxX
WINDOWS_IMG=/dev/disk/by-partuuid/xxxxxxxx-XX
...
    -drive index=0,media=disk,file=$WINDOWS_IMG,format=raw \
...


Convert the VM to UEFI

Microsoft assumes you will not be using a UEFI BIOS with the VM. But we
need a UEFI BIOS for passthrough to avoid the
issues with using Seabios. Since version 1703, Windows 10 includes
the MBR2GPT command line tool to convert a drive from MBR to GPT and to
create the EFI partition. The following is based on instructions at
Windows Central

Boot Windows with SeaBIOS.
$ vim windows_seabios
#!/bin/sh

WINDOWS_IMG=/dev/sdxX

qemu-system-x86_64 \
    -enable-kvm \
    -machine type=q35,accel=kvm \
    -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_vendor_id=NoFortyThree \
    -m 4G \
    -soundhw hda \
    -usb \
    -drive index=0,media=disk,file=$WINDOWS_IMG,format=raw \
    -monitor stdio

$ chmod +x windows_seabios
$ xsudo ./windows_seabios

Log in with the password "Passw0rd!".

Open Settings. Search for "Advanced startup" and choose "Change advanced startup options". Click Restart now.

Choose Troubleshoot > Advanced options > Command prompt.
Wait a long time.

When prompted, select "IEUser" and enter the password "Passw0rd!".

At the command prompt, type:
> mbr2gpt /validate
> mbr2gpt /convert


Exit the command prompt and choose Turn off your PC.

Boot the VM using the windows VM script. If you are confronted
with the UEFI shell, enter ...
> exit

... to return to the BIOS. Choose Boot manager and select the
hard drive Windows is installed on.


Configuring NVIDIA in Windows


Use Chocolatey for software management. You can find instructions at Chocolatey Docs.

Install the NVIDIA driver. Don't time out if it takes ages to download.
> choco install -y --execution-timeout=0 nvidia-display-driver


Install Steam.
> choco install -y steam


Optimising Windows


Assign more processors to Windows. nproc will tell you how many processors are available. e.g.
$ nproc
8
$ vim windows
...
    -smp 4 \
...


Use virtio SCSI drivers. Previous versions of Windows allowed you to use the Recovery Drive to set the boot drive to use virtio drivers, but Windows 10 will throw "stop code: inaccessible boot device", and requires a longer process. Credit: harrymc and llegolas.

Download Windows VirtIO drivers from RedHat:
$ wget https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso


Set a second drive as a virtio SCSI drive. Add the virtio driver CD.
$ vim windows
...
# /dev/sdd3
DATA_IMG=/dev/disk/by-partuuid/c6a2a56c-2e66-4642-9785-c18cae539c0f
...
    -device virtio-scsi-pci,id=scsi0 \
    -drive id=disk1,file=$DATA_IMG,format=raw,cache=writeback,if=none \
      -device scsi-hd,drive=disk1 \
    -drive index=2,media=cdrom,file=virtio-win.iso \
...


Boot Windows. Open Device Manager. Right-click on "PCI SCSI Controller" and select "Update driver". Load the driver from D:vioscsiw10amd64.

Run Command Prompt as administrator and set Windows to boot into safe mode:
> bcdedit /set {current} safeboot minimal


Shut down Windows and change the boot device type to virtio:
$ vim windows
...
    -device virtio-scsi-pci,id=scsi0 \
    -drive id=disk0,file=$WINDOWS_IMG,format=raw,cache=none,if=none \
      -device scsi-hd,drive=disk0 \
    -drive id=disk1,file=$DATA_IMG,format=raw,cache=writeback,if=none \
      -device scsi-hd,drive=disk1 \
...


Start the VM. Windows will enter in safe mode.

Note
In Safe mode all boot-start drivers will be enabled and loaded, including the virtio driver. Since there is now a drive installed to use it, the kernel will now make it part of the drivers that are to be loaded on boot and not disable it again.


Run Command Prompt as administrator again, and set Windows no longer to boot into safe mode with:
> bcdedit /deletevalue {current} safeboot


Reboot.

Further reading:

Tuning VM Disk Performance
> The example below shows the use of the virtio-scsi-pci driver. In this case I defined an ioh3420 root port driver (PCIe) where I attached the SCSI devices:
-device pcie-root-port,bus=pcie.0,addr=1c.0,multifunction=on, port=1,chassis=1,id=root.1 \
-object iothread,id=io1 \
-device virtio-scsi-pci,id=scsi0,iothread=io1,num_queues=4,bus=pcie.0 \
-drive id=scsi0,file=/dev/sdb1,if=none,format=raw,aio=threads,cache=none \
-device scsi-hd,drive=scsi0 \


Use virtio network drivers. Note that if VM's usage is high, this seems to be able to cripple network speed for the host.

Load the vhost-net kernel module, and set the Ethernet controller to virtio
$ vim windows
...
modprobe vhost-net
...
    -netdev user,id=net0 \
        -device virtio-net,netdev=net0 \
...


Boot Windows. Open Device Manager. Right-click on "Ethernet Controller" and choose "Update driver". Load the driver from D:NetKVMw10amd64.


Disable Hibernate: Run "cmd" as administrator, and
> powercfg -h off


Disable Suspend: Start > Settings > System > Power & sleep > Sleep: "Never"

Disable Cortana:

Run "gpedit.msc"
Go to Computer Configuration > Administrative Templates > Windows Components > Search
Find "Allow Cortana", and set it to "Disabled". Click "OK".
Reboot, or log out and log in.


You can tweak the audio quality with QEMU_PA_SAMPLES and QEMU_AUDIO_TIMER_PERIOD. You can find the available variables and their defaults with:
$ qemu-system-x86_64 -audio-help


Use NAT networking with a TAP interface.
Further reading:

https://en.wikibooks.org/wiki/QEMU/Networking#TAP_interfaces
https://wiki.qemu.org/Documentation/Networking/NAT
https://wiki.archlinux.org/index.php/QEMU#Networking
https://wiki.gentoo.org/wiki/QEMU#Host_configuration


If you are going to assign more than 4GB of RAM to Windows, hugepage
support will improve performance. hugepages is installed in Ubuntu
by default. The following is based on Ubuntu Community Help Wiki
and ArchWiki.

Confirm that hugepage size is 2048 KB:
$ cat /proc/meminfo | grep Hugepagesize
Hugepagesize:       2048 kB


If we want to assign 6 GB to Windows, that will be
6 × 1024 × 1024 ÷ 2048 = 6 × 1024 ÷ 2 = 3072 hugepages. Round up
to 3100. If we want to assign 12 GB to Windows, that will be
12 × 1024 ÷ 2 = 6144 hugepages. Round up to 6200. Add the
following to the script to reserve 3100 (for example) hugepages:
$ vim windows
...
sysctl vm.nr_hugepages=3100
...
    -m 6G \
    -mem-path /dev/hugepages \
...


You can check, while the VM is running, how many pages are used:
$ cat /proc/meminfo | grep HugePages


CPU pinning:


Further reading:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#CPU_pinning


References


QEMU User Documentation


How to take back the NVIDIA

If you want to use the NVIDIA in Linux again, use the following steps to take the graphics card back:

Comment out "pci_stub" in /etc/initramfs-tools/modules

Move /lib/modprobe.d/pci-stub.conf to ~/doc/
$ sudo mv /lib/modprobe.d/pci-stub.conf ~/doc/


Update the initramfs image:
$ sudo update-initramfs -u


Reboot.

If necessary, check prime graphics device and monitor configuration:
$ xsudo nvidia-settings
$ rm ~/.config/monitors.xml

Reboot, log in, and configure monitors.