nvidia_driver_mismatch_without_rebooting.md

NVML: Driver/library version mismatch

Need to unload the the Nvidia driver kernel module (kmod) and then load it again as the current version is wrong.

Which drivers are loaded?

lsmod | grep nvidia

nvidia_uvm            634880  8
nvidia_drm             53248  0
nvidia_modeset        790528  1 nvidia_drm
nvidia              12312576  86 nvidia_modeset,nvidia_uvm

We need to unload nvidia mod, so first unload everything that it depends on:

sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia_uvm

then, unload nvidia

sudo rmmod nvidia

Troubleshooting

The previous steps will most likely result in ERROR: Module nvidia is in use, which indicates that the kernel module is in use, you should kill the process that using the kmod:

sudo lsof /dev/nvidia*

Kill the processes:

sudo kill <PID>

Confirm you can successfully unload those kmods:

lsmod | grep nvidia

The output of the above command should be empty.

Then run (should work):

nvidia-smi

To make sure there is only one version of nvidia-drivers installed:

$ dpkg -l | grep -i nvidia

ii  libnvidia-ifr1-450:amd64                   450.51.05-0ubuntu1                               amd64        NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  nsight-compute-2020.1.1                    2020.1.1.8-1                                     amd64        NVIDIA Nsight Compute
ii  nvidia-compute-utils-450                   450.51.05-0ubuntu1                               amd64        NVIDIA compute utilities
ii  nvidia-dkms-450                            450.51.05-0ubuntu1                               amd64        NVIDIA DKMS package
ii  nvidia-driver-450                          450.51.05-0ubuntu1                               amd64        NVIDIA driver metapackage
ii  nvidia-kernel-common-450                   450.51.05-0ubuntu1                               amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-450                   450.51.05-0ubuntu1                               amd64        NVIDIA kernel source package
ii  nvidia-modprobe                            450.51.05-0ubuntu1                               amd64        Load the NVIDIA kernel driver and create device files
ii  nvidia-prime                               0.8.8.2                                          all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                            450.51.05-0ubuntu1                               amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-450                           450.51.05-0ubuntu1                               amd64        NVIDIA driver support binaries
ii  xserver-xorg-video-nvidia-450    

Remove everything that is not required (all libraries with 440, 390 etc):

sudo apt-get remove --purge nvidia-381 nvidia-450

Let ubuntu pick the best driver:

sudo ubuntu-drivers devices
sudo ubuntu-drivers autoinstall