In the name of God
This gist contains steps to setup Ubuntu 22.04 for deep learning.
- Computer name: Name-PC
- Name: Name
- User name: name
- Password: ********
$ sudo apt update
$ sudo apt full-upgrade --yes
$ sudo apt autoremove --yes
$ sudo apt autoclean --yes
$ reboot- Create a file (
~/full-update.sh) with the following lines:
#!/usr/bin/env bash
if [ "$EUID" -ne 0 ]
then echo "Error: Please run as root."
exit
fi
clear
echo "################################################################################"
echo "Updating list of available packages..."
echo "--------------------------------------------------------------------------------"
apt update
echo "################################################################################"
echo
echo "################################################################################"
echo "Upgrading the system by removing/installing/upgrading packages..."
echo "--------------------------------------------------------------------------------"
apt full-upgrade --yes
echo "################################################################################"
echo
echo "################################################################################"
echo "Removing automatically all unused packages..."
echo "--------------------------------------------------------------------------------"
apt autoremove --yes
echo "################################################################################"
echo
echo "################################################################################"
echo "Clearing out the local repository of retrieved package files..."
echo "--------------------------------------------------------------------------------"
apt autoclean --yes
echo "################################################################################"
echo- Review Ubuntu Settings
- Review Software & Updates
$ sudo ~/full-update.sh
Install Chrome (https://www.google.com/chrome)
$ sudo apt install ./google-chrome-stable_current_amd64.deb
$ sudo apt install build-essential pkg-config cmake ninja-build
$ sudo apt install python3 python3-venv python3-pip python3-setuptools python3-wheel python3-dev
$ sudo apt install git
$ git config --global user.name "Name"
$ git config --global user.email "name@domain.com"
$ git config --global core.editor "gedit -s"- Copy your own SSH keys to
~/.ssh/
Check Display Hardware:
$ sudo lshw -C display
Install NVIDIA GPU Driver:
- Install from GUI: Software & Updates > Additional Drivers > NVIDIA
Try $ sudo ubuntu-drivers autoinstall if NVIDIA drivers are disabled.
You can also install it from the terminal: $ sudo apt install nvidia-driver-535.
Check TensorFlow and CUDA Compatibilities:
Install CUDA Toolkit (CUDA 11.8):
- Install prerequisites:
$ sudo apt install linux-headers-$(uname -r)
- Download CUDA 11.8 (https://developer.nvidia.com/cuda-toolkit-archive)
$ wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
- Install CUDA 11.8:
$ sudo ./cuda_11.8.0_520.61.05_linux.run --override(without Driver) - Set up the development environment by modifying the
PATHandLD_LIBRARY_PATHvariables (Add following lines to~/.bashrc):export PATH=$PATH:/usr/local/cuda-11.8/binexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.8/lib64:/usr/local/cuda-11.8/extras/CUPTI/lib64
- Check that GPUs are visible using the following command (A reboot may be required):
$ nvidia-smi
Install cuDNN v8.6, for CUDA 11.8:
- https://developer.nvidia.com/cudnn
- https://developer.nvidia.com/rdp/cudnn-archive
- Login & Download: https://developer.nvidia.com/compute/cudnn/secure/8.6.0/local_installers/11.8/cudnn-local-repo-ubuntu2204-8.6.0.163_1.0-1_amd64.deb
$ sudo dpkg -i cudnn-local-repo-ubuntu2204-8.6.0.163_1.0-1_amd64.deb$ sudo cp /var/cudnn-local-repo-ubuntu2204-8.6.0.163/cudnn-local-FAED14DD-keyring.gpg /usr/share/keyrings/$ sudo apt update$ sudo apt install libcudnn8$ sudo apt install libcudnn8-dev$ sudo apt install libcudnn8-samples # Optional
Reboot:
$ reboot
$ python3 -m venv ~/venvs/ml
$ source ~/venvs/ml/bin/activate
(ml) $ pip install --upgrade pip setuptools wheel
(ml) $ pip install --upgrade numpy scipy matplotlib ipython jupyter pandas sympy nose
(ml) $ pip install --upgrade scikit-learn scikit-image
(ml) $ deactivate$ python3 -m venv ~/venvs/tfcpu
$ source ~/venvs/tfcpu/bin/activate
(tfcpu) $ pip install --upgrade pip setuptools wheel
(tfcpu) $ pip install --upgrade tensorflow-cpu tensorboard keras
(tfcpu) $ deactivate$ python3 -m venv ~/venvs/tfgpu
$ source ~/venvs/tfgpu/bin/activate
(tfgpu) $ pip install --upgrade pip setuptools wheel
(tfgpu) $ pip install --upgrade tensorflow tensorboard keras
(tfgpu) $ deactivateVerification:
$ source ~/venvs/tfgpu/bin/activate
(tfgpu) $ python
>>> from tensorflow.python.client import device_lib
>>> device_lib.list_local_devices()
>>> exit()
(tfgpu) $ deactivate$ python3 -m venv ~/venvs/torchcpu
$ source ~/venvs/torchcpu/bin/activate
(torchcpu) $ pip install --upgrade pip setuptools wheel
(torchcpu) $ pip install --upgrade torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu
(torchcpu) $ deactivate$ python3 -m venv ~/venvs/torchgpu
$ source ~/venvs/torchgpu/bin/activate
(torchgpu) $ pip install --upgrade pip setuptools wheel
(torchgpu) $ pip install --upgrade torch torchvision torchaudio
(torchgpu) $ deactivateVerification:
$ source ~/venvs/torchgpu/bin/activate
(torchgpu) $ python
>>> import torch
>>> torch.cuda.is_available()
>>> exit()
(torchgpu) $ deactivate$ python3 -m venv ~/venvs/fastai
$ source ~/venvs/fastai/bin/activate
(fastai) $ pip install --upgrade pip setuptools wheel
(fastai) $ pip install --upgrade fastai fastbook
(fastai) $ deactivate$ python3 -m venv ~/venvs/onnx
$ source ~/venvs/onnx/bin/activate
(onnx) $ pip install --upgrade pip setuptools wheel
(onnx) $ pip install --upgrade onnx onnxruntime-gpu
(onnx) $ deactivateVerification:
$ source ~/venvs/onnx/bin/activate
(onnx) $ python
>>> import onnxruntime as ort
>>> ort.get_device()
>>> exit()
(onnx) $ deactivate$ python3 -m venv ~/venvs/qt
$ source ~/venvs/qt/bin/activate
(qt) $ pip install --upgrade pip setuptools wheel
(qt) $ pip install --upgrade PySide6
(qt) $ deactivate- https://conda.io/
- https://docs.conda.io/en/latest/miniconda.html
- https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html
- https://anaconda.org/search
Install:
$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ chmod +x Miniconda3-latest-Linux-x86_64.sh
$ ./Miniconda3-latest-Linux-x86_64.sh
$ # Do you wish the installer to initialize Miniconda3
# by running conda init?
# yes
$ source ~/miniconda3/bin/activate
$ conda config --set auto_activate_base false
$ conda deactivateActivate and Deactivate:
$ conda activate
(base) $ conda deactivateManaging conda:
(base) $ conda info
(base) $ conda update condaManaging environments:
(base) $ conda info --envs
(base) $ conda create --name snakes python=3.5
(base) $ conda info --envs
(base) $ conda activate snakes
(snakes) $ python --version
(snakes) $ conda search beautifulsoup4
(snakes) $ conda install beautifulsoup4
(snakes) $ conda list
(snakes) $ conda update beautifulsoup4
(snakes) $ conda uninstall beautifulsoup4
(snakes) $ conda list
(snakes) $ conda deactivate
(base) $ conda remove --name snakes --all
(base) $ conda info --envs- Project documentation with Markdown:
- https://www.mkdocs.org/
- https://squidfunk.github.io/mkdocs-material/
- https://code.visualstudio.com
sudo apt install ./<file>.deb
Install Python extension for Visual Studio Code:
Recommended Extensions:
- Python: https://marketplace.visualstudio.com/items?itemName=ms-python.python
- Pylance: https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance
- Python Debugger: https://marketplace.visualstudio.com/items?itemName=ms-python.debugpy
- Jupyter: https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter
- Black Formatter: https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter
- isort: https://marketplace.visualstudio.com/items?itemName=ms-python.isort
- Docker: https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker
- Dev Containers: https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers
- Remote - SSH: https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh
- Code Spell Checker: https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker
Other Recommended Extensions:
- C/C++ Extension Pack: https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools-extension-pack
- Makefile Tools: https://marketplace.visualstudio.com/items?itemName=ms-vscode.makefile-tools
- ESLint: https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint
- Prettier - Code formatter: https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode
Enable GPU support for PyCharm Projects:
- Edit Configurations...
- Environment variables:
PATH=$PATH:/usr/local/cuda-11.8/binLD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.8/lib64:/usr/local/cuda-11.8/extras/CUPTI/lib64
- https://www.qt.io/
- https://doc.qt.io/qt-6/linux.html
$ sudo apt install build-essential libgl1-mesa-dev
Nvidia Container Toolkit:
- https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/overview.html
- https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#docker
- https://hub.docker.com/r/nvidia/cuda
Make sure you have installed the NVIDIA driver and Docker engine for your Linux distribution Note that you do not need to install the CUDA Toolkit on the host system, but the NVIDIA driver needs to be installed.
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
$ sudo apt update
$ sudo apt-get install --yes nvidia-container-toolkit
$ sudo nvidia-ctk runtime configure --runtime=docker
$ sudo systemctl restart docker
# NOTE: --runtime=nvidia
$ docker container run --rm --runtime=nvidia nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
# OR
$ docker container run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smiTensorFlow Docker:
# CPU
$ docker run -it tensorflow/tensorflow:latest bash
# GPU
$ docker run --gpus all -it tensorflow/tensorflow:latest-gpu bashPyTorch Docker:
$ docker run --gpus all --rm -ti --ipc=host pytorch/pytorch:latestRunning ARM Docker Containers:
- https://en.wikipedia.org/wiki/QEMU
- https://en.wikipedia.org/wiki/Binfmt_misc
- https://wiki.debian.org/QemuUserEmulation
- https://github.com/multiarch/qemu-user-static
- https://github.com/docker-library/official-images#architectures-other-than-amd64
$ sudo apt install qemu qemu-user-static binfmt-support
$ # Host Architecture: x86_64
$ uname -m
$ # ARM Container Architecture: armv7l
$ docker run --rm arm32v7/debian uname -m$ sudo apt install ubuntu-restricted-extras
$ sudo apt install virtualbox virtualbox-dkms virtualbox-ext-pack virtualbox-guest-additions-iso
$ sudo apt install curl wget uget tar zip unzip rar unrar
$ sudo apt install gimp vlc ffmpeg
$ sudo apt install kdiff3- Ubuntu Software > System Load Indicator
$ mkdir ~/.fonts- Copy fonts to
~/.fonts
Yeah I can't get CUDA to set up correctly. Running
nvidia-smiafter foregoing the installation of the driver just yields NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running., as is to be expected I guess