CUDA-ready Archlinux for WSL2

cuda-ready-archlinux-for-wsl2.md

This is a brief guide on how to install Archlinux as a WSL2 distribution and how to set up CUDA afterwards.

As of late, Window's WSL2 offers GPU passthrough from WSL2/Linux to Windows for NVidia graphics cards which allows to run (and develop) CUDA-based applications on the WSL2/Linux-side with almost native performance. Unfortunately, the official guides for the CUDA setup for WSL2/Linux are predominantly Ubuntu-specific. Here's to you, Arch!

1. Install Archlinux

0. Make sure that your Windows meets the dependencies and that your WSL2 is set up. See these instructions.

Archlinux is not among the default distributions available for WSL2. We'll install it from a tarball instead, a functionality offered natively by the WSL.

1. Download the latest release of the root-fs tarball from the Archlinux Gitlab. Pick the base-devel version whose filename at the time of writing is base-devel-20211124.0.39890.tar.xz.

The WSL can work with pure .tar archives or its g-zip compressed counterparts, .tar.gz archives, but it cannot handle .tar.xz archives.

2. Generate a .tar archive from the downloaded .tar.xz file by decompressing.

Under Linux this can be done via unxz -c "$your_file.tar.xz" > arch_base-devel.tar. If you don't have quick access to a Linux machine, install the WSL2 Ubuntu distribution from the Microsoft Store. The Windows filesystem is mounted at /mnt/c/, whence you can access and decompress the downloaded archive.

We now need to set up a space in the Windows filesystem where all the data for the Archlinux distribution will reside.

3. Create a new directory for the Archlinux distribution, e.g C:\wsl-distributions\archlinux.

We're now ready to install Archlinux from the tarball.

4. In a PowerShell execute wsl --import archlinux C:\wsl-distributions\archlinux C:\Path-to-your-tar-file\arch_base-devel.tar.

Once this is finished, your Archlinux distribution is ready to roll. Open a new Windows Terminal and launch a new Archlinux shell. You may now configure the system to your liking. In order to not have to maintain Docker build scripts on Windows and WSL2/Linux, the Docker Desktop Windows application offers WSL2 integration.

5. Install the Docker Desktop Windows application and enable the WSL2 engine.

Done. You may want to explore the WSL configuration parameters and adjust your setup. With regards to this guide this step is completely optional and can be postponed.

Disclaimer:

• There's no systemd support for WSL2 yet, so don't install software which depends on it.
• You may be unable to ping as a normal user. Run sudo setcap cap_net_raw+p /bin/ping to fix the issue. See this issue for more information.

2. Install CUDA

Luckily for Archlinux users, the CUDA installation is trivial because the cuda and cuda-tools packages don't explicitly depend on the Linux NVidia drivers, which must not be installed as they would interfere with the WSL2's GPU "paravirtualization".

1. In Windows install the latest CUDA-enabled GPU drivers for WSL from the NVidia Developer Page.
2. In Archlinux Install the cuda and cuda-tools packages: pacman -S cuda cuda-tools.

You should now be ready to write and use CUDA-based applications and software. You can test it by building a simple example from the NVidia CUDA Introduction or run a CUDA-based Docker container from the NVidia WSL User Guide.

Disclaimer:

• There's no WSL2 support for the CUDA profiling tools yet.
• Some old tutorials use nvidia-docker, which is obsolete as Docker now natively handles NVidia GPUs. Instead of nvidia-docker run simply use docker run --gpus all. If you must use nvidia-docker, which is available from the AUR, and you run into trouble with Docker, try running the Docker daemon on the Linux side and disable Docker Desktop.

Resources

• https://docs.microsoft.com/en-us/windows/wsl/use-custom-distro
• https://docs.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl
• https://docs.nvidia.com/cuda/wsl-user-guide/index.html

Hi @slavistan, while I tried to install cuda, the package requires nvidia-utils-510.60.02-1, which do contains an extra libcuda.so and seems like a driver (not sure).

Have you tried if that will conflict with the Windows version under /usr/lib/wsl/lib?

The detailed output of pacman is shown below:

$ sudo pacman -S cuda
resolving dependencies...
looking for conflicting packages...
warning: dependency cycle detected:
warning: harfbuzz will be installed before its freetype2 dependency
warning: dependency cycle detected:
warning: libglvnd will be installed before its nvidia-utils dependency
warning: dependency cycle detected:
warning: mesa will be installed before its libglvnd dependency

Packages (52) egl-wayland-2:1.1.9+r3+g582b2d3-1  eglexternalplatform-1.1-2  freetype2-2.12.0-2  graphite-1:1.3.14-1
              harfbuzz-4.2.0-3  libdrm-2.4.110-1  libedit-20210910_3.1-1  libepoxy-1.5.10-1  libevdev-1.12.1-1
              libfontenc-1.1.4-3  libglvnd-1.4.0-1  libgudev-237-1  libinput-1.20.1-1  libomxil-bellagio-0.9.3-3
              libpciaccess-0.16-2  libpng-1.6.37-3  libunwind-1.6.2-1  libwacom-2.2.0-1  libx11-1.7.5-1  libxau-1.0.9-3
              libxcb-1.14-1  libxcvt-0.1.1-1  libxdamage-1.1.5-3  libxdmcp-1.1.3-3  libxext-1.3.4-3  libxfixes-6.0.0-1
              libxfont2-2.0.5-1  libxkbfile-1.1.0-2  libxrandr-1.5.2-3  libxrender-0.9.10-4  libxshmfence-1.3-2
              libxxf86vm-1.1.4-4  llvm-libs-13.0.1-2  lm_sensors-1:3.6.0.r41.g31d1f125-1  mesa-22.0.1-3  mtdev-1.1.6-1
              nvidia-utils-510.60.02-1  opencl-nvidia-510.60.02-1  pixman-0.40.0-1  vulkan-icd-loader-1.3.208-1
              wayland-1.20.0-2  xcb-proto-1.14.1-5  xf86-input-libinput-1.2.1-1  xkeyboard-config-2.35.1-1
              xorg-fonts-encodings-1.0.5-2  xorg-server-21.1.3-6  xorg-server-common-21.1.3-6  xorg-setxkbmap-1.3.3-1
              xorg-xkbcomp-1.4.5-1  xorg-xrandr-1.5.1-2  xorgproto-2021.5-1  cuda-11.6.1-1

Total Download Size:   1622.50 MiB
Total Installed Size:  4842.56 MiB

Thanks if you got any idea.

Hi @RichardYCJ. I didn't have any trouble with it, although I must admit that the fact that we have multiple libcuda.so files flying around did escape me. Did you try it out yourself?

Should there be any problems we could try to symlink /usr/lib/libcuda.so to the Windows version, or, alternatively, we could try to fiddle with the load order of libraries. In the end, the cleanest solution would of course be to create a custom package (cuda-wsl) which doesn't install libcuda at all but I'd postpone that until we know for certain that we'll need it.

I reckon you just give it a try as is.

Hi @slavistan, did a little search and now I would say you're right.

the cuda and cuda-tools packages don't explicitly depend on the Linux NVidia drivers.

What nvidia-utils-510.60.02-1 means should be some cuda libraries built upon drive version 510.60.02. Essentially cuda is just a toolkit, or I would say general framework, that requires GPU driver to run.

While installing, I got some error about the libcuda in wsl

ldconfig: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

and I solved it by using mklink in the CMD to manually link it to libcuda.so.1.1

After the installation, my nvidia-smi (now from /usr/sbin) can not display the GPU memory correctly. And that is due to my outdated drive version. After I upgrade to Game ready 512.15, everything works like a charm.

To sum up:

• Do what the article says about Step 1
• Match your GPU drive version in Windows with the arch repo (now is 510.60), it may retain binary compatibility between minor versions
• Check your libcuda.so to be a symbol link, convert manually if necessary
• Install cuda, cuda-tools and you are good to go

FYI, I use Windows 11 with build 22000.613

@RichardYCJ I just stumbled upon this and I'm having a similar error. Could you clarify what you mean by converting libcuda.so to a symbolic link? I don't think I understand enough about what's happening under the hood here to follow the instructions. mklink makes me think I'm supposed to do something on the Windows side, but I'm not clear what.

Thanks very much.

Not sure @ianepreston but I think is talking about microsoft/WSL#5663 (comment)

So far I've been able to just install cuda and cuda-tools in the WSL environment and have it work. It gives that error still, but I didn't have to do anything with mklink.

Setting LD_LIBRARY_PATH="/usr/lib/wsl/lib:$LD_LIBRARY_PATH" works if you find something is trying to load /usr/lib/libcuda.so. This is less destructive then overriding the /usr/lib/libcuda.so symlink.

Example running hashcat on ArchWSL:

LD_LIBRARY_PATH="/usr/lib/wsl/lib:$LD_LIBRARY_PATH" hashcat -a 0 -m 100 ./hash.txt ./rockyou.txt --backend-ignore-opencl