Setup for XRDP using Nvidia Acceleration

xrdp-nvidia-setup.sh

#!/usr/bin/env bash

set -e

cd ~

sudo -v

# Make sure system is in a good, updated, clean, state.
sudo apt-get -y update
sudo apt-get -y dist-upgrade
sudo apt-get -y autoremove

# Install CUDA (Shamelessly copied from Nvidia's website):
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda

sudo usermod $USER -a -G video
sudo usermod $USER -a -G tty
sudo usermod $USER -a -G render

# XRDP Build Pre-reqs Part 1
sudo apt-get -y install git autoconf libtool pkg-config gcc g++ make \
    libssl-dev libpam0g-dev libjpeg-dev libx11-dev libxfixes-dev libxrandr-dev \
    flex bison libxml2-dev intltool xsltproc xutils-dev python3-libxml2 g++ \
    xutils libfuse-dev libmp3lame-dev nasm libpixman-1-dev xserver-xorg-dev \
    libjson-c-dev libsndfile1-dev libspeex-dev libspeexdsp-dev libpulse-dev \
    libpulse0 autopoint \*turbojpeg\* libfdk-aac-dev libopus-dev libgbm-dev \
    libx264\* libx264-dev
 
#XRDP Build Pre-reqs Part 2 (For some reason apt needs this to be separate)
sudo apt-get -y install libepoxy-dev
 
#System Pre-reqs
sudo apt-get -y install xorg xserver-xorg-legacy dbus-x11
sudo apt-get -y remove dbus-user-session

sudo tee /etc/X11/Xwrapper.config > /dev/null << EOL
# Xwrapper.config (Debian X Window System server wrapper configuration file)
#
# This file was generated by the post-installation script of the
# xserver-xorg-legacy package using values from the debconf database.
#
# See the Xwrapper.config(5) manual page for more information.
#
# This file is automatically updated on upgrades of the xserver-xorg-legacy
# package *only* if it has not been modified since the last upgrade of that
# package.
#
# If you have edited this file but would like it to be automatically updated
# again, run the following command as root:
#   dpkg-reconfigure xserver-xorg-legacy 
needs_root_rights=no
allowed_users=anybody
EOL

echo "gnome-session --session=ubuntu" > ~/.xsession
sudo chmod +x ~/.xsession
sudo tee ~/.xsessionrc > /dev/null << EOL
export DESKTOP_SESSION=ubuntu
export GNOME_SHELL_SESSION_MODE=ubuntu
export XDG_CURRENT_DESKTOP=ubuntu:GNOME
EOL
sudo chmod +x ~/.xsessionrc

BUILD_DIR=$(mktemp -d)

echo "Building xrdp..."
git clone https://github.com/Nexarian/xrdp.git --branch mainline_merge "$BUILD_DIR/xrdp"
cd "$BUILD_DIR/xrdp"
./bootstrap
./configure --enable-fuse --enable-rfxcodec --enable-pixman --enable-mp3lame \
    --enable-sound --enable-opus --enable-fdkaac --enable-x264 --enable-nvenc
make -j $(nproc) clean all
sudo make install

echo "Building xorgxrdp..."
git clone https://github.com/Nexarian/xorgxrdp.git --branch mainline_merge "$BUILD_DIR/xorgxrdp"
cd "$BUILD_DIR/xorgxrdp"
./bootstrap
./configure --with-simd --enable-lrandr
make -j $(nproc) clean all
sudo make install

echo "Configuring Nvidia..."
BUS_ID=$(nvidia-smi --query-gpu=pci.bus --format=csv | sed -n '2 p' | xargs -I{} printf "%d\n" {})
# Alternative method to get the BUS ID that doesn't work on some systems (Such as a laptop with an Intel iGPU and an Nvidia Discrete GPU).
# BUS_ID=$(lspci | grep -e 'VGA.*NVIDIA' | cut -d: -f1 | xargs -I{} printf "%d\n" 0x{} | sed -n 1p)
# nvidia-xconfig --query-gpu-info
sudo sed -i -E 's/(BusID "PCI:)[[:digit:]]+(:0:0")/\1'$BUS_ID'\2/' /etc/X11/xrdp/xorg_nvidia.conf

echo "Starting the server..."
sudo systemctl enable xrdp
sudo service xrdp start

I'm trying to run xrdp with gpu acceleration on a headless ubuntu 22.04 server. It is a fresh installation and I have done apt install ubuntu-desktop and then run this script. But the desktop does not load, I get "Can't create session for user testusr - X server could not be started".

What is missing, what am I doing wrong?

logs

As far as I know, if you want a headless server, you have to have a seated session and then configure your Xorg session with a virtual display card. I haven't tried yet, but I have tested it with Sunshine server. I will post the Sunshine server setup and you can adapt it to xrdp. If it works, you can also re-post a xrdp version to help others.

Setup

NOTE: The steps here only need to be performed once on your computer; after these are done, you only need to follow the steps under the "Execution" section.

1. sudo systemctl set-default multi-user.target (original graphical.target): Set the graphical target to multi-user target so that the server will not give desktop manager on boot. After done, reboot your system.

2. sudo usermod -a -G tty myusername: The user that will be connecting in via SSH must be in the 'tty' group on the system where Sunshine will be started. This is required because when we start Xorg, it will attempt to access the specific virtual TTY we give it to use. NOTE: You'd better reboot to take effect. Use the 'groups' command to verify that you see 'tty' before proceeding.

3. sudo setcap CAP_SYS_RAWIO+eip /usr/lib/xorg/Xorg: You need to run the command below to give the capability 'CAP_SYS_RAWIO+eip' to the /usr/lib/xorg/Xorg executable. Note that some more research needs to be done into the implications of doing this but if you don't provide that capability than the Xorg server session will fail with an error: xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted)

4. sudo vim /etc/X11/Xwrapper.conf: You will need to edit the /etc/X11/Xwrapper.conf file and change the 'allowed_users' to 'anybody'. While this will have some security implications they should be managable as this should only be giving the required permission to valid and logged in users to start an Xorg server session. The default, which is typically 'console', requires that you log in with one of the virtual TTYs for the system and this is no different than physically logging into the system anyways (the exact thing we're trying to avoid). Reboot your system to make sure this takes effect before proceeding.

5. You will need to create a custom Xorg configuration file that is suitable for use with a "headless" session (i.e., one in which we don't really use the physically attached monitor for). To do this follow the steps below:

   1. SSH into the machine where you want to run Sunshine.
   2. vim ~/xorg.conf: Create a new xorg.conf file with the following content:

         Section "ServerLayout"
             Identifier "TwinLayout"
             Screen 0 "metaScreen" 0 0
         EndSection
         
         Section "Monitor"
             Identifier "Monitor0"
             Option "Enable" "true"
         EndSection
         
         Section "Device"
             Identifier "Card0"
             Driver "nvidia"
             VendorName "NVIDIA Corporation"
             Option "MetaModes" "1920x1080"
             Option "ConnectedMonitor" "DP-0"
             Option "ModeValidation" "NoDFPNativeResolutionCheck,NoVirtualSizeCheck,NoMaxPClkCheck,NoHorizSyncCheck,NoVertRefreshCheck,NoWidthAlignmentCheck"
         EndSection
         
         Section "Screen"
             Identifier "metaScreen"
             Device "Card0"
             Monitor "Monitor0"
             DefaultDepth 24
             Option "TwinView" "True"
             SubSection "Display"
                 Modes "1920x1080"
             EndSubSection
         EndSection
      

Execution

NOTE: The steps here need to be performed each time that you want to bring up your Sunshine server. Note that the way the steps are given the entire setup (Xorg server, Sunshine, and desktop session) will be running under the context of your SSH session so once you exit the terminal all these things will exit. You will need to leave your SSH session up while you're using your Sunshine server. An alternative would be to use screen so that these processes stay up after you exit your terminal connection.

1. SSH into the machine where you want to bring up the Sunshine server.
2. Decide which virtual TTY (1-5) you want to tie the Xorg session to based on which are unused (we assume we use tty2). Note that TTYs 1-5 should correspond to the virtual teletype consoles that you can drop into if you were to use the key sequence [Ctrl] + [Alt] + [F1] (or [F2], etc).
3. sudo chmod g+r /dev/tty2: Add read access to the group on its file descriptor.
4. sudo Xorg -noreset +extension GLX +extension RANDR +extension RENDER vt2 -logfile /home/YOURUSERNAME/X_LOG.txt -config /home/YOURUSERNAME/xorg.conf :9 >> /home/YOURUSERNAME/std_stream.out 2>&1 &: Bring up an Xorg server and run it in the shell background.
5. export DISPLAY=":9": In your shell, set the DISPLAY variable you specified when bringing up the Xorg server. This number is set in the previous command where [:x] is the number.
6. sunshine &: Start Sunshine in the background.
7. exec gnome-session or exec startplasma-x11: Start your desktop environment session.

Hope the above works!!!

References:
https://chadrick-kwag.medium.com/configuring-x-server-on-headless-server-a8b5c3bc7d9
https://unix.stackexchange.com/questions/503078/start-headless-x-server-as-non-root-user
https://www.howtogeek.com/428174/what-is-a-tty-on-linux-and-how-to-use-the-tty-command/
https://unix.stackexchange.com/questions/159124/xf86enableioports-failed-to-set-iopl-for-i-o-operation-not-permitted

@alexanderzjs
Thanks for your post.
I got it working on my local test machine with xserver-xorg-video-dummy and by adding a monitor to xorg_nvidia.conf.
It still does not work on the server, but I think it is a GPU/cuda/driver issue.

@alexanderzjs Thanks for your post. I got it working on my local test machine with xserver-xorg-video-dummy and by adding a monitor to xorg_nvidia.conf. It still does not work on the server, but I think it is a GPU/cuda/driver issue.

Great! Adding a virtual monitor is definitely the solution.

Hi @Nexarian, thank you for the Nvidia GPU setup. It works for me. However, I get 2 Xorg and gnome-shell processes at the same time when checking nvidia-smi. Is that normal? Is it any possible to optimize this one? Many thanks.

Really want to try it on openSUSE Tumbleweed on KDE Plasma Wayland but I'm too dumb and (still) cannot adapt the procedures to my distribution/desktop manager =(

Unfortunately XRDP is not compatible with wayland, and likely won't be for some time.

Unfortunately XRDP is not compatible with wayland, and likely won't be for some time.

Thanks.
I'll try to change desktop manager but I still don't know if I'll be able to adapt the steps to openSUSE. I'll try to google it.

I have been trying for a while to get this working on a headless machine. I think I am close (I can see the Xorg server and the window manager in nvidia-smi), however get the below errors about process group ID in the sesman.log file.

[INFO ] Starting the xrdp channel server for display :10
[ERROR] Can't set process group ID of 135797 to 135782 [Operation not permitted]
[INFO ] Session in progress on display :10. Waiting until the window manager (pid 135797) exits to end the session
[ERROR] Can't set process group ID of 135798 to 135782 [Operation not permitted]

and then eventually it says there was an error connecting to the session. The windows remote desktop looks like this. Any help would be appreciated, I've spent a few days on this so far. I'm on an ubuntu 22.04 vmware virtual machine server with a nvidia A40 gpu. Unfortunately my work is restrictive in that I can't copy and paste any logs directly.

I have been trying for a while to get this working on a headless machine. I think I am close (I can see the Xorg server and the window manager in nvidia-smi), however get the below errors about process group ID in the sesman.log file.

[INFO ] Starting the xrdp channel server for display :10
[ERROR] Can't set process group ID of 135797 to 135782 [Operation not permitted]
[INFO ] Session in progress on display :10. Waiting until the window manager (pid 135797) exits to end the session
[ERROR] Can't set process group ID of 135798 to 135782 [Operation not permitted]

and then eventually it says there was an error connecting to the session. The windows remote desktop looks like this. Any help would be appreciated, I've spent a few days on this so far. I'm on an ubuntu 22.04 vmware virtual machine server with a nvidia A40 gpu. Unfortunately my work is restrictive in that I can't copy and paste any logs directly.

You probably have to tell the steps you are following. Otherwise, it is quite hard to give you a direction. Did you add remote login user to "tty" and "video" groups? Did you try "sudo chmod g+r /dev/tty2" to add read access to the group on its file descriptor?

You probably have to tell the steps you are following. Otherwise, it is quite hard to give you a direction. Did you add remote login user to "tty" and "video" groups? Did you try "sudo chmod g+r /dev/tty2" to add read access to the group on its file descriptor?

Apologies, you are right. I will have to give more details. Yes, I did both of those things. I also added the user to event because I was getting some other errors. I followed pretty much exactly the original gist, as well as the relevant steps from @alexanderzjs's post. I will try to post more detail when I'm back at work on Monday, I don't remember exactly every change I made, but I noted them down.