peppergrayxyz/qemu-vulkan-virtio.md

## qemu-vulkan-virtio.md

      
    Raw
  

              qemu-vulkan-virtio.md
            
          
    QEMU with VirtIO GPU Vulkan Support

With its latest reales qemu added the Venus patches so that virtio-gpu now support venus encapsulation for vulkan. This is one more piece to the puzzle towards full Vulkan support.
An outdated blog post on clollabora described in 2021 how to enable 3D acceleration of Vulkan applications in QEMU through the Venus experimental Vulkan driver for VirtIO-GPU with a local development environment. Following up on the outdated write up, this is how its done today.
Definitions

Let's start with the brief description of the projects mentioned in the post & extend them:

QEMU is a machine emulator
VirGL is an OpenGL driver for VirtIO-GPU, available in Mesa.
Venus is an experimental Vulkan driver for VirtIO-GPU, also available in Mesa.
Virglrenderer is a library that enables hardware acceleration to VM guests, effectively translating commands from the two drivers just mentioned to either OpenGL or Vulkan.
libvirt is an API for managing platform virtualization
virt-manager is a desktop user interface for managing virtual machines through libvirt

Merged Patches:

2024-08-14 venus: make cross-device optional merged in mesa 24.2
2024-11-25 KVM: Stop grabbing references to PFNMAP'd pages merged in linux 6.13
2024-11-12 Support blob memory and venus on qemu merged in qemu 9.2.0

Work in progress:

libvirt Add support for more virtio-vga-gl arguments #638
virt-manager Add support for Venus / Vulkan VirtIO-GPU driver #362

Prerequisites

Make sure you have the proper version installed on the host:

linux kernel >= 6.13 built with CONFIG_UDMABUF
working Vulkan and kvm setup
qemu >= 9.2.0
virglrenderer with enabled venus support
mesa >= 24.2.0

You can verify this like so:
$ uname -r
6.13.0
$ ls /dev/udmabuf
/dev/udmabuf
$ ls /dev/kvm
/dev/kvm
$ qemu-system-x86_64 --version
QEMU emulator version 9.2.0
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

Check your distros package sources how they build virglrenderer.
For Vulkan to work you need the proper drivers to be installed for your graphics card.
To verfiy your setup, install vulkan-tools. Make sure mesa >= 24.2.0 and test vkcube:
$ vulkaninfo --summary | grep driverInfo
	driverInfo         = Mesa 24.2.3-1ubuntu1
	driverInfo         = Mesa 24.2.3-1ubuntu1 (LLVM 19.1.0)
...
$ vkcube
Selected GPU x: ..., type: ...

Building qemu

If your distro doesn't (yet) ship and updated version of qemu, you can build it yourself from source:
wget https://download.qemu.org/qemu-9.2.0.tar.xz
tar xvJf qemu-9.2.0.tar.xz
cd qemu-9.2.0
mkdir build && cd build
../configure --target-list=x86_64-softmmu  \
  --enable-kvm                 \
  --enable-opengl              \
  --enable-virglrenderer       \
  --enable-gtk                 \
  --enable-sdl
make -j4

The configuration step will throgh errors if packages are missing.
Check the qemu wiki for further info what to install: https://wiki.qemu.org/Hosts/Linux
Create and run an image for QEMU

Create an image & fetch the distro of your choice:
Host

ISO=ubuntu-24.10-desktop-amd64.iso  
wget https://releases.ubuntu.com/oracular/ubuntu-24.10-desktop-amd64.iso  

IMG=ubuntu-24-10.qcow2
qemu-img create -f qcow2 $IMG 16G
Run a live version or install the distro
qemu-system-x86_64                                               \
    -enable-kvm                                                  \
    -M q35                                                       \
    -smp 4                                                       \
    -m 4G                                                        \
    -cpu host                                                    \
    -net nic,model=virtio                                        \
    -net user,hostfwd=tcp::2222-:22                              \
    -device virtio-vga-gl,hostmem=4G,blob=true,venus=true        \
    -vga none                                                    \
    -display gtk,gl=on,show-cursor=on                            \
    -usb -device usb-tablet                                      \
    -object memory-backend-memfd,id=mem1,size=4G                 \
    -machine memory-backend=mem1                                 \
    -hda $IMG                                                    \
    -cdrom $ISO                                                  

Adjust the parameters accordingly:

smp: number of cpu cores
m: RAM
hostmem,size: VRAM

Guest

Install mesa-utilites and vulkan-tools to test the setup:
$ glxinfo -B

$ vkcube
Selected GPU x: ..., type: ...

If the deive is llvmpipe somehting is wrong. The device should be virgl (...).
Troubleshooting


(host) add -d guest_errors to show error messages from the guest
(guest) try installing vulkan virtio drivers and mesa
check the original blog post

Ubuntu 24.10

This is how you do it on Ubuntu
kernel

Install mainline: https://github.com/bkw777/mainline
sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline

find the latest kernel (>= 6.13), at the time of writing 6.13 is a release candidate, so include those:
$ mainline check --include-rc
Install kernel:
$ sudo mainline install 6.13-rc1
Verfify installed kernels:
$ mainline list-installed
mainline 1.4.10
Installed Kernels:
linux-image-6.11.0-13-generic
linux-image-generic-hwe-24.04
linux-image-unsigned-6.13.0-061300rc1-generic
mainline: done

reboot into new kernel
verify running kernel
$ uname -r
6.13.0-061300rc1-generic

virglrenderer

the ubuntu package is not compiled with the proper flags.
If installed remove it: $ sudo apt-get remove libvirglrenderer-dev
download, build & install from source with venus enabled
wget    https://gitlab.freedesktop.org/virgl/virglrenderer/-/archive/1.1.0/virglrenderer-1.1.0.tar.gz
sudo apt-get install python3-full ninja-build libvulkan-dev libva-dev
python3 -m venv venv
venv/bin/pip install meson
venv/bin/meson build -Dvideo=true -Dvenus=true
ninja -C build
ninja install

qemu

install qemu >= 9.2.0, at the time of writing ubuntu has not yet packaged it
Install build depdencies: https://wiki.qemu.org/Hosts/Linux
sudo apt-get install build-essential pip libslirp-dev slirp
sudo apt-get install git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev ninja-build
sudo apt-get install git-email
sudo apt-get install libaio-dev libbluetooth-dev libcapstone-dev libbrlapi-dev libbz2-dev
sudo apt-get install libcap-ng-dev libcurl4-gnutls-dev libgtk-3-dev
sudo apt-get install libibverbs-dev libjpeg8-dev libncurses5-dev libnuma-dev
sudo apt-get install librbd-dev librdmacm-dev
sudo apt-get install libsasl2-dev libsdl2-dev libseccomp-dev libsnappy-dev libssh-dev
sudo apt-get install libvde-dev libvdeplug-dev libvte-2.91-dev libxen-dev liblzo2-dev
sudo apt-get install valgrind xfslibs-dev 
sudo apt-get install libnfs-dev libiscsi-dev

build and run as described
virt-manager

-- work in progress --
Currently this is work in progress, so there is no option to add vulkan support in virt-manager. There are no fields to configure this. Also xml doesnt work, because libvirt doesn't know about these options either, so xml validation fails. There is however an option for QEMU command-line passthrough which bypasses the validation.
If you setup a default machine with 4G of memory, you can do this:
  <qemu:commandline>
    <qemu:arg value="-device"/>
    <qemu:arg value="virtio-vga-gl,hostmem=4G,blob=true,venus=true"/>
    <qemu:arg value="-object"/>
    <qemu:arg value="memory-backend-memfd,id=mem1,size=4G"/>
    <qemu:arg value="-machine"/>
    <qemu:arg value="memory-backend=mem1"/>
    <qemu:arg value="-vga"/>
    <qemu:arg value="none"/>
  </qemu:commandline>
Which gives this error:
qemu-system-x86_64: virgl could not be initialized: -1

Changing the number from 4G to 4194304k (same as memory) leds to this error:
qemu-system-x86_64: Spice: ../spice-0.15.2/server/red-qxl.cpp:435:spice_qxl_gl_scanout: condition `qxl_state->gl_draw_cookie == GL_DRAW_COOKIE_INVALID' failed

to be further investigated.