In the following gist I'm going to guide you through the process of installing and booting an entire linux distribution with full desktop environment just like you would have with a classical VM, but with much better performance and much worse isolation :)
The reason why I did this was mainly because it's cool, but also to test new distros with decent graphics performance without actually booting them on my PC.
If you "try this at home" just keep in mind a container is not as secure as a VM, and some of the option we're going to explore will weaken container isolation from "a bit risky" to "totally unsafe" depending on what you choose.
Also, we're going to use systemd-nspawn for containers as it's probably the best fit for our use case and can also boot any linux partition without needing to prepare an apposite container image.
Less go!
First off, you will need to...
There are at least three options I can think of:
- Bootstrap the distro of your choice in a folder with the appropriate tool
- Boot and install the distro of your choice in a Virtualbox VM, then extract the VM filesystem to a folder
- Just boot a distro from another partition
Let's go with the former.
For this example I chose Ubuntu 22.04 but you can install anything you like (Arch, Fedora, Ubuntu, Debian, ...) - just have a look at man systemd-nspawn for more details.
- Install
debootstrapandschroot(the name of some packages might change from distro to distro) mkdir ubuntu2204sudo debootstrap --arch amd64 jammy ubuntu2204/ http://archive.ubuntu.com/ubuntu/- Chroot into the system:
sudo systemd-nspawn -D ubuntu2204 - Create your user:
adduser username - Add your user to the following groups:
sudo usermod -aG video,render,sudo username - Name resolution is probably broken due to systemd-resolved that can't run at this time, so
mv /etc/resolv.conf{,.old} && echo "nameserver 8.8.8.8" > /etc/resolv.conf- of course you can change this to your preferred DNS. - Configure lang and locale:
dpkg-reconfigure locales && apt install language-pack-it && locale-gen - Remove
acpidbecause of an Ubuntu 22.04 bug..apt remove acpid - Then
exitthe container
You will need to install your host gpu drivers to get good performance.
- Boot the container:
sudo systemd-nspawn -D ubuntu2204/ -b --register=no --keep-unit --bind=/dev/dri - Install some deps:
sudo apt install libglvnd-dev pkg-config
If you're an nvidia user like me, you'll probably be better off installing the same driver version you have on your host system from the nvidia website and do not install kernel modules:
sudo bash NVIDIA-Linux-*.run --no-kernel-modules will do.
If you're an AMD or INTEL user, I don't know what the procedure will be, but I'm going to guess it'll just work if you install your drivers like you normally do :)
You could skip this step depending on what configuration you end up setting to your container, but it's strongly recommended
to install VirtualGL as it's going to be useful to get 3D graphics out of your container in the more "safe" configuration.
- Boot into the container:
sudo systemd-nspawn -D ubuntu2204/ -b --register=no --keep-unit --bind=/dev/dri
- Install dependencies and useful stuff:
sudo add-apt-repository universe
sudo apt install libegl1-mesa mesa-utils mesa-utils-bin
- Install virtualgl from sourceforge (as it's probably not going to be in your repos, unless you use arch btw)
wget https://deac-ams.dl.sourceforge.net/project/virtualgl/3.0.1/virtualgl_3.0.1_amd64.deb
sudo dpkg -i virtualgl*.deb
- Configure VirtualGL:
sudo vglserver_config
- Press 1 to Configure server for use with VirtualGL
Answer 'Yes' to Restrict 3D X server access to vglusers group
Answer 'Yes' to Restrict framebuffer device access to vglusers group
Answer 'Yes' to Disable XTEST extension
Press X to Exit
- Add the user to the
vglusersgroup.
sudo usermod -aG vglusers username
- Add this to your
/etc/environmentfile:
PULSE_SERVER=tcp:127.0.0.1 # to hear audio from the container
DISPLAY=:12 # must match your Xephyr display id as we'll see later
- If you're an NVIDIA user, also add those env. variables to for the correct renderer:
__NV_PRIME_RENDER_OFFLOAD=1
__GLX_VENDOR_LIBRARY_NAME=nvidia
__VK_LAYER_NV_optimus=NVIDIA_only
VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json
If you are an AMD or Intel user you'll probably need to do something similar but I can't test that for you. Have a look at this page for more info and let me know what works for you!
- Now you can
sudo rebootthe container
Now to the fun part. Here you have multiple options from "relatively safe" with acceptable performance to "very unsafe" with good performance. From my benchmarks, there is a ~30% gap in performance between the safer and the riskiest option.
Is to run everything in a Xephyr window, not sharing memory or resources with the container directly.
Performance is still quite good, but on the lower end of what you can get if you risk more as we'll see later. Still, I think this should be the preferred option for most people.
- Run a Xephyr (X11 windowed server) session on the display you previously set:
Xephyr :12 -screen 1920x1080 & - Optionally give anonymous local access to your
pulse/pipewireaudio serverpactl load-module module-native-protocol-tcp auth-anonymous=1 auth-ip-acl="127.0.0.1"(not recommended if you don't need it) - Boot the container
sudo systemd-nspawn -D ubuntu2204 -b --register=no --keep-unit
- make sure you've disabled X11 shared memory as it won't work:
export QT_X11_NO_MITSHM=1
- launch the Desktop Environment
vlgrun -d :12 gnome-session
PS: You can add QT_X11_NO_MITSHM=1 to your /etc/environment to make this permanent across reboots.
There are a few middleground options between this solution and the safer one, but I'm not sure they really make much sense since we're breaking container isolation anyways. Needless to say you'll need to be very sure of what you're running when using this method, as it will be just as risky as running on your own machine and OS from a security perspective.
So... to get the most out of your container you'll need to share memory and devices directly with it.
SYSTEMD_NSPAWN_SHARE_NS_IPC is a dangerous (and therefore undocumented?) environment variable for systemd-nspawn which will let the host and the container share IPC namespace. When using this, make sure QT_X11_NO_MITSHM=0 is set. Last thing you need to do is to share the video card with the container for EGL access.
Wrapping everything up, we get..
pactl load-module module-native-protocol-tcp auth-anonymous=1 auth-ip-acl="127.0.0.1" # for audio
Xephyr :12 -screen 1920x1080 &
SYSTEMD_NSPAWN_SHARE_NS_IPC=1 sudo -E systemd-nspawn -D ubuntu2204/ -b --register=no --keep-unit --bind=/dev/shm --bind=/dev/dri
export QT_X11_NO_MITSHM=0
vglrun -d /dev/dri/card0 gnome-session # for EGL
OR
vglrun -d :12 gnome-session # for GLX
OR
vglrun -d :0 application # to run an application in seamless mode with the host*
*To be fair this shouldn't work unless you do xhost +local: which is a bad idea, or via a proper authentication which is still not a great idea.
I hope I didn't forget anything important but let me know if so. I'll leave you with a couple
You might want to read to get a better feeling of what you're doing:
Linux graphics stack: https://blogs.igalia.com/itoral/2014/07/29/a-brief-introduction-to-the-linux-graphics-stack/
Linux graphics stack and drivers: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/inside-linux-graphics-paper.pdf
Diving into Mesa: https://blogs.igalia.com/itoral/2014/08/08/diving-into-mesa/
VirtualGL background: https://virtualgl.org/About/Background
VirtualGL documentation: https://rawcdn.githack.com/VirtualGL/virtualgl/3.0.1/doc/index.html
LD_PRELOAD: https://www.baeldung.com/linux/ld_preload-trick-what-is
Hi! I've followed your guide with Debian 11 as guest and Debian Testing as host but it seems that something's broken.
glxinforeports the following (regardless of whether it's run byvglrunor not, in both the safe and unsafe setups):Do you have an idea about what's going on?