Setup podman and nvidia-container-runtime

README.md

How to setup nvidia-container-runtime and podman/runc

Podman is such a cool project! However, there is no easy way to setup the nvidia-container-runtime and podman so that we can run unprivileged container in a gpu host. This is specially interesting in environments with multiple people accessing the same host (strong isolation between containers!!!).

Steps to setup the whole system

1. Install podman and friends (buildah and skopeo)

   Ubuntu: add-apt-repository -y ppa:projectatomic/ppa && apt install podman buildah skopeo

   Fedora: yum install podman buildah skopeo

2. Install the nvidia-container-runtime Instructions in https://nvidia.github.io/libnvidia-container/

3. Install the nvidia-hook.json

cat <<EOF >> /usr/share/containers/oci/hooks.d/oci-nvidia-hook.json
{
  "hook": "/usr/bin/nvidia-container-runtime-hook",
  "arguments": ["prestart"],
  "annotations": ["sandbox"],
  "stage": [ "prestart" ]
}
EOF

4. Configure the nvidia-container-runtime

cat <<EOF >> /etc/nvidia-container-runtime/config.toml
disable-require = false

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-runtime-hook.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
no-cgroups = true
#user = "root:video"
ldconfig = "@/sbin/ldconfig.real"
EOF

How to use the system?

Just use the podman as before, but it the nvidia-container-runtime-hook will inject the nvidia driver into the container runtime. It Just Works™

podman run -it --rm nvidia/cuda nvidia-smi
Tue May  7 14:09:49 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.56       Driver Version: 418.56       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce MX150       Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   33C    P8    N/A /  N/A |      0MiB /  2002MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+