Replacing cudnn libraries on ubuntu linux for improved performance on 4090 GPUs

instructions.txt

Inspiration: https://www.reddit.com/r/StableDiffusion/comments/y71q5k/4090_cudnn_performancespeed_fix_automatic1111/

This document describes how to replace the libcudnn libs in an Ubuntu environment to improve the performance
of pytorch with 4090 GPUs.

Follow the standard nvidia cudatoolkit installation instructions using apt (do NOT use the runfile,
these instructions may not work).  Instructions here: https://developer.nvidia.com/cuda-downloads

ensure your installation is functional.  The following command should list your driver and cuda versions as well
as your specific GPU information.   

> nvidia-smi

Now we are ready to swap the lib files! 

find the libcudnn package:

> apt list --installed | grep cudnn
>
> libcudnn8-dev/unknown,now 8.9.0.131-1+cuda12.1 amd64 [installed]
> libcudnn8/unknown,now 8.9.0.131-1+cuda12.1 amd64 [installed]

find the location of the files for this package:

> dpkg -L libcudnn8
> /.
> /usr
... (omitted)
> /usr/lib/x86_64-linux-gnu/libcudnn.so.8
> /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8
> /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8
> /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8
> /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8
> /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8
> /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8

from within your venv folder find the corresponding lib files:

> find . -type f -name "libcudnn*"
> ./lib/python3.10/site-packages/nvidia/cudnn/lib/
... (omitted)

Optionally backup your existing lib files.

Replace these lib files with the ones from the apt package:

> cp usr/lib/x86_64-linux-gnu/libcudnn* ./lib/python3.10/site-packages/nvidia/cudnn/lib/