Setup for XRDP using Nvidia Acceleration

xrdp-nvidia-setup.sh

#!/usr/bin/env bash

set -e

cd ~

sudo -v

# Make sure system is in a good, updated, clean, state.
sudo apt-get -y update
sudo apt-get -y dist-upgrade
sudo apt-get -y autoremove

# Install CUDA (Shamelessly copied from Nvidia's website):
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda

sudo usermod $USER -a -G video
sudo usermod $USER -a -G tty
sudo usermod $USER -a -G render

# XRDP Build Pre-reqs Part 1
sudo apt-get -y install git autoconf libtool pkg-config gcc g++ make \
    libssl-dev libpam0g-dev libjpeg-dev libx11-dev libxfixes-dev libxrandr-dev \
    flex bison libxml2-dev intltool xsltproc xutils-dev python3-libxml2 g++ \
    xutils libfuse-dev libmp3lame-dev nasm libpixman-1-dev xserver-xorg-dev \
    libjson-c-dev libsndfile1-dev libspeex-dev libspeexdsp-dev libpulse-dev \
    libpulse0 autopoint \*turbojpeg\* libfdk-aac-dev libopus-dev libgbm-dev \
    libx264\* libx264-dev
 
#XRDP Build Pre-reqs Part 2 (For some reason apt needs this to be separate)
sudo apt-get -y install libepoxy-dev
 
#System Pre-reqs
sudo apt-get -y install xorg xserver-xorg-legacy dbus-x11
sudo apt-get -y remove dbus-user-session

sudo tee /etc/X11/Xwrapper.config > /dev/null << EOL
# Xwrapper.config (Debian X Window System server wrapper configuration file)
#
# This file was generated by the post-installation script of the
# xserver-xorg-legacy package using values from the debconf database.
#
# See the Xwrapper.config(5) manual page for more information.
#
# This file is automatically updated on upgrades of the xserver-xorg-legacy
# package *only* if it has not been modified since the last upgrade of that
# package.
#
# If you have edited this file but would like it to be automatically updated
# again, run the following command as root:
#   dpkg-reconfigure xserver-xorg-legacy 
needs_root_rights=no
allowed_users=anybody
EOL

echo "gnome-session --session=ubuntu" > ~/.xsession
sudo chmod +x ~/.xsession
sudo tee ~/.xsessionrc > /dev/null << EOL
export DESKTOP_SESSION=ubuntu
export GNOME_SHELL_SESSION_MODE=ubuntu
export XDG_CURRENT_DESKTOP=ubuntu:GNOME
EOL
sudo chmod +x ~/.xsessionrc

BUILD_DIR=$(mktemp -d)

echo "Building xrdp..."
git clone https://github.com/Nexarian/xrdp.git --branch mainline_merge "$BUILD_DIR/xrdp"
cd "$BUILD_DIR/xrdp"
./bootstrap
./configure --enable-fuse --enable-rfxcodec --enable-pixman --enable-mp3lame \
    --enable-sound --enable-opus --enable-fdkaac --enable-x264 --enable-nvenc
make -j $(nproc) clean all
sudo make install

echo "Building xorgxrdp..."
git clone https://github.com/Nexarian/xorgxrdp.git --branch mainline_merge "$BUILD_DIR/xorgxrdp"
cd "$BUILD_DIR/xorgxrdp"
./bootstrap
./configure --with-simd --enable-lrandr
make -j $(nproc) clean all
sudo make install

echo "Configuring Nvidia..."
BUS_ID=$(nvidia-smi --query-gpu=pci.bus --format=csv | sed -n '2 p' | xargs -I{} printf "%d\n" {})
# Alternative method to get the BUS ID that doesn't work on some systems (Such as a laptop with an Intel iGPU and an Nvidia Discrete GPU).
# BUS_ID=$(lspci | grep -e 'VGA.*NVIDIA' | cut -d: -f1 | xargs -I{} printf "%d\n" 0x{} | sed -n 1p)
# nvidia-xconfig --query-gpu-info
sudo sed -i -E 's/(BusID "PCI:)[[:digit:]]+(:0:0")/\1'$BUS_ID'\2/' /etc/X11/xrdp/xorg_nvidia.conf

echo "Starting the server..."
sudo systemctl enable xrdp
sudo service xrdp start

Hi @Nexarian, thank you for the Nvidia GPU setup. It works for me. However, I get 2 Xorg and gnome-shell processes at the same time when checking nvidia-smi. Is that normal? Is it any possible to optimize this one? Many thanks.

Really want to try it on openSUSE Tumbleweed on KDE Plasma Wayland but I'm too dumb and (still) cannot adapt the procedures to my distribution/desktop manager =(

Unfortunately XRDP is not compatible with wayland, and likely won't be for some time.

Unfortunately XRDP is not compatible with wayland, and likely won't be for some time.

Thanks.
I'll try to change desktop manager but I still don't know if I'll be able to adapt the steps to openSUSE. I'll try to google it.

I have been trying for a while to get this working on a headless machine. I think I am close (I can see the Xorg server and the window manager in nvidia-smi), however get the below errors about process group ID in the sesman.log file.

[INFO ] Starting the xrdp channel server for display :10
[ERROR] Can't set process group ID of 135797 to 135782 [Operation not permitted]
[INFO ] Session in progress on display :10. Waiting until the window manager (pid 135797) exits to end the session
[ERROR] Can't set process group ID of 135798 to 135782 [Operation not permitted]

and then eventually it says there was an error connecting to the session. The windows remote desktop looks like this. Any help would be appreciated, I've spent a few days on this so far. I'm on an ubuntu 22.04 vmware virtual machine server with a nvidia A40 gpu. Unfortunately my work is restrictive in that I can't copy and paste any logs directly.

I have been trying for a while to get this working on a headless machine. I think I am close (I can see the Xorg server and the window manager in nvidia-smi), however get the below errors about process group ID in the sesman.log file.

[INFO ] Starting the xrdp channel server for display :10
[ERROR] Can't set process group ID of 135797 to 135782 [Operation not permitted]
[INFO ] Session in progress on display :10. Waiting until the window manager (pid 135797) exits to end the session
[ERROR] Can't set process group ID of 135798 to 135782 [Operation not permitted]

and then eventually it says there was an error connecting to the session. The windows remote desktop looks like this. Any help would be appreciated, I've spent a few days on this so far. I'm on an ubuntu 22.04 vmware virtual machine server with a nvidia A40 gpu. Unfortunately my work is restrictive in that I can't copy and paste any logs directly.

You probably have to tell the steps you are following. Otherwise, it is quite hard to give you a direction. Did you add remote login user to "tty" and "video" groups? Did you try "sudo chmod g+r /dev/tty2" to add read access to the group on its file descriptor?

You probably have to tell the steps you are following. Otherwise, it is quite hard to give you a direction. Did you add remote login user to "tty" and "video" groups? Did you try "sudo chmod g+r /dev/tty2" to add read access to the group on its file descriptor?

Apologies, you are right. I will have to give more details. Yes, I did both of those things. I also added the user to event because I was getting some other errors. I followed pretty much exactly the original gist, as well as the relevant steps from @alexanderzjs's post. I will try to post more detail when I'm back at work on Monday, I don't remember exactly every change I made, but I noted them down.