with Ubuntu 18.04 and higher, this might be useful: https://askubuntu.com/questions/61396/how-do-i-install-the-nvidia-drivers/61433#61433
However, it seems that contrary to what the ubuntu-drivers devices | grep nvidia recommends, for Ubuntu 16.04, installing the latest recommended version of the driver (currently 430) does not play nice with X (maybe my kernal is not updated). And once I update the kernal I'll have to go through this all over again. So, better write shit down.
For now, following this
NOTE: May remove CUDA installations as well*
sudo apt-get purge nvidia*
sudo apt-get autoremove
sudo apt-get install nvidia-375 nvidia-modprobe
Check pre-installation steps at NVIDIA docs.
Finally download the toolkit at http://developer.nvidia.com/cuda-downloads.
For installing CUDA 10.1 at Ubuntu 16.04, x86_64:
Installer used: deb(network)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-ubuntu1604.pin
sudo mv cuda-ubuntu1604.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda
Add exports for environment variables to bashrc:
export PATH=/usr/local/cuda-10.1/bin:/usr/local/cuda-10.1/NsightCompute-2019.1${PATH:+:${PATH}}
export LD_LIBRARY_PATH="/usr/local/cuda-10.1/lib64:${LD_LIBRARY_PATH+${LD_LIBRARY_PATH}}"
Finally move CUDA out from root into home and create symlink to avoid path issues