Skip to content

Instantly share code, notes, and snippets.

@FrankieWOO

FrankieWOO/install nvidia driver with real-time kernel.md

Last active April 29, 2024 22:43
Show Gist options
  • Save FrankieWOO/8f5d0794512ae43a4efd7e068b19a4d8 to your computer and use it in GitHub Desktop.
Save FrankieWOO/8f5d0794512ae43a4efd7e068b19a4d8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
install nvidia driver with real-time kernel.md

How to install Nvidia driver alongside real-time linux kernel

To install Nvidia driver on a PC running real-time kernel, follow the tutorial here: https://github.com/ApolloAuto/apollo/blob/master/docs/howto/how_to_install_apollo_kernel.md

https://github.com/cacao-org/cacao/wiki/OS-install-RTC

The rt kernel I tested at this moment is 5.15.21-rt30 (Feb 2022).

Install real-time linux kernel

Follow a community contributed tutorial to install the latest stable RT_PREEMPT version https://docs.ros.org/en/foxy/Tutorials/Building-Realtime-rt_preempt-kernel-for-ROS-2.html

Open Software & Updates. in the Ubuntu Software menu tick the ‘Source code’ box.

Install dependencies:

sudo apt-get update
sudo apt-get build-dep linux
sudo apt-get install build-essential bc curl ca-certificates gnupg2 lsb-release libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf fakeroot
sudo apt update
sudo apt install zstd
cd ~
mkdir rt_kernel
cd rt_kernel

wget https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.15.21.tar.gz
wget https://mirrors.edge.kernel.org/pub/linux/kernel/v5.x/linux-5.15.21.tar.sign
wget https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.15/older/patch-5.15.21-rt30.patch.gz
wget https://cdn.kernel.org/pub/linux/kernel/projects/rt/5.15/older/patch-5.15.21-rt30.patch.sign

gunzip linux-5.15.21.tar.gz
# tar -xzf linux-5.15.21.tar.gz

gunzip patch-5.15.21-rt30.patch.gz

Verify the kernel files integrity

gpg2 --verify linux-*.tar.sign
gpg2 --verify patch-*.patch.sign
tar xf linux-*.tar

cd linux-*/
patch -p1 < ../patch-*.patch

cp /boot/config-$(uname -r) .config

yes '' | make oldconfig
make menuconfig

apply these modifications:

  • general setup / preemption model
    • Fully preemptible Kernel (Real-time)
  • general setup / Timers subsystem
    • High Resolution Timer Support
  • general setup / Timers subsystem / Timer tick handling
    • Full dynticks system (tickless)
  • Processor type and features / Timer frequency
    • 1000 Hz
  • Power management and ACPI options
    • CPU Frequency scaling, CPU Frequency scaling (CPU_FREQ [=y]) -> Default CPUFreq governor ( [=y]) (X) performance
  • Cryptographic API* > Certificates for signature checking (at the very bottom of the list) > Provide system-wide ring of trusted keys > Additional X.509 keys for default system keyring
    • Remove the “debian/canonical-certs.pem” from the prompt and press Ok.

save and exit.

make -j$(nproc) deb-pkg

or

make -j $(nproc)
sudo make bzImage
sudo make INSTALL_MOD_STRIP=1 modules_install -j $(nproc)
sudo make install

According to the post by cacao:

the INSTALL_MOD_STRIP is important - it shrinks the initial ramdisk size by ~90%. In some cases, when not used, the initial ramdisk would not load and the patched kernel would not boot.

If you see these errors:

  • if you see this error: CONFIG_X86_X32 enabled but no binutils support, change config_x86_x32 to n
  • sed: can't read modules.order: No such file or directory
    • set CONFIG_SYSTEM_TRUSTED_KEY=""
    • set CONFIG_SYSTEM_REVOCATION_KEYS=""
  • Missing file: arch/x86/boot/bzImage , you need to run sudo make bzImage before modules_install
  • if you see this error:bin/sh: 1: zstd: not found, you need to install Zstandard bysudo apt install zstd

Make sure realtime kernel is set to default

Edit /etc/default/grub file Change GRUB_DEFAULT=0 to GRUB_DEFAULT=saved. Add GRUB_SAVEDEFAULT=true

Allow a user to set real-time permissions for its processes

According to https://frankaemika.github.io/docs/installation_linux.html#setting-up-the-real-time-kernel:

sudo addgroup realtime
sudo usermod -a -G realtime $(whoami)

Afterwards, add the following limits to the realtime group in /etc/security/limits.conf:

@realtime soft rtprio 99
@realtime soft priority 99
@realtime soft memlock 102400
@realtime hard rtprio 99
@realtime hard priority 99
@realtime hard memlock 102400

Install CUDA and nvidia driver

Install CUDA

Run the following to install CUDA 11.3

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.3.0/local_installers/cuda-repo-ubuntu2004-11-3-local_11.3.0-465.19.01-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-3-local_11.3.0-465.19.01-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-3-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

The corresponding driver version is 465 465 failed

sudo apt-get install nvidia-driver-470

Install Nvidia driver for realtime kernel

Run the following to build Nvidia driver 470

sudo apt-get install nvidia-kernel-source-470
# Change to Nvidia driver source directory /usr/src/nvidia-465.19.01
cd "$(dpkg -L nvidia-kernel-source-470 | grep -m 1 "nvidia-drm" | xargs dirname)"

# Build Nvidia driver with IGNORE_PREEMPT_RT_PRESENCE=1
sudo env NV_VERBOSE=1 \
    make -j$(nproc) NV_EXCLUDE_BUILD_MODULES='' \
    KERNEL_UNAME=$(uname -r) \
    IGNORE_XEN_PRESENCE=1 \
    IGNORE_CC_MISMATCH=1 \
    IGNORE_PREEMPT_RT_PRESENCE=1 \
    SYSSRC=/lib/modules/$(uname -r)/build \
    LD=/usr/bin/ld.bfd \
    modules

sudo mv *.ko /lib/modules/$(uname -r)/updates/dkms/
sudo depmod -a

reboot the system

sudo reboot

run nvidia-smi to check the driver

option 2

A guide how to install nvidia driver manually: https://www.if-not-true-then-false.com/2021/debian-ubuntu-linux-mint-nvidia-guide/

Install dependencies

sudo apt-get update -y
sudo apt-get install -y libglvnd-dev
# 1. Download NVIDIA driver as a .run file

# 2. Stop X-Server
sudo service lightdm stop

# 3. Blacklist Nouveau driver
sudo nano /etc/modprobe.d/blacklist-nouveau.conf

# Insert into file:
#  blacklist nouveau
#  options nouveau modeset=0

# 4. Update kernel initramfs
sudo update-initramfs -u
sudo reboot  # I'm not sure if needed

# 5. Install driver!
chmod +x NVIDIA-Linux-*.run
sudo IGNORE_PREEMPT_RT_PRESENCE=1 bash NVIDIA-Linux-*.run  # Insert downloaded .run file

# 6. Reboot
sudo reboot

https://gist.github.com/pantor/9786c41c03a97bca7a52aa0a72fa9387

if the nvidia-installer create modprobe files to disable Nouveau driver, and if you want to re-enable the Nouveau driver later, you will need to delete these files:

/usr/lib/modprobe.d/nvidia-installer-disable-nouveau.conf
/etc/modprobe.d/nvidia-installer-disable-nouveau.conf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment