Skip to content

Instantly share code, notes, and snippets.

@Titus-von-Koeller
Last active June 28, 2024 15:49
Show Gist options
  • Save Titus-von-Koeller/e541c1175b0a191bac75b72d9964e9d0 to your computer and use it in GitHub Desktop.
Save Titus-von-Koeller/e541c1175b0a191bac75b72d9964e9d0 to your computer and use it in GitHub Desktop.
{ pkgs, unstable }:
let
cudatoolkit = pkgs.cudaPackages.cudatoolkit;
nixLD = pkgs.runCommand "ld.so" {} ''
mkdir -p $out/lib64
ln -s ${pkgs.glibc}/lib64/ld-linux-x86-64.so.2 $out/lib64/ld-linux-x86-64.so.2
'';
nixLDLibraryPath = pkgs.lib.makeSearchPath "lib" [
pkgs.addOpenGLRunpath.driverLink
pkgs.glib.out
pkgs.libglvnd
pkgs.stdenv.cc.cc.lib
];
in
pkgs.buildFHSUserEnv {
name = "pixi-env";
targetPkgs = pkgs: with pkgs; [
unstable.pixi
cudatoolkit
glib
libglvnd
stdenv.cc.cc.lib
# bashInteractive
# bash-completion
# complete-alias
];
runScript = "bash";
profile = ''
export NVIDIA_DRIVER_CAPABILITIES=compute,utility
export XDG_CONFIG_DIRS=/run/opengl-driver/share/X11/xorg.conf.d''${XDG_CONFIG_DIRS:+:}$XDG_CONFIG_DIRS
export XDG_DATA_DIRS=/run/opengl-driver/share''${XDG_DATA_DIRS:+:}$XDG_DATA_DIRS
export LD_LIBRARY_PATH=${cudatoolkit}/lib:${cudatoolkit}/lib64:${cudatoolkit}/lib64/stubs:/run/opengl-driver/lib:${pkgs.glib}/lib:${pkgs.libglvnd}/lib:${pkgs.stdenv.cc.cc.lib}/lib''${LD_LIBRARY_PATH:+:}$LD_LIBRARY_PATH
export CUDA_PATH=${cudatoolkit}
export PATH=${cudatoolkit}/bin:$PATH
export LIBRARY_PATH=${cudatoolkit}/lib:${cudatoolkit}/lib64:$LIBRARY_PATH
export CPLUS_INCLUDE_PATH="${cudatoolkit}/include''${CPLUS_INCLUDE_PATH:+:$CPLUS_INCLUDE_PATH}"
export C_INCLUDE_PATH="${cudatoolkit}/include''${C_INCLUDE_PATH:+:$C_INCLUDE_PATH}"
export NIX_LD=${nixLD}/lib64/ld-linux-x86-64.so.2
export NIX_LD_LIBRARY_PATH=${nixLDLibraryPath}
# Pixi completion -- not working yet, due to missing `complete` command
eval "$(pixi completion --shell bash 2>/dev/null)"
echo "*** Pixi environment activated, using $(which pixi). ***"
'';
}
@Titus-von-Koeller
Copy link
Author

configuration.nix

[...]
  boot.kernelModules = [ "nvidia_uvm" ];  # <-- is this really needed?
  [...]
  
  hardware.opengl = {
    enable = true;
    driSupport = true;
    driSupport32Bit = true;
  };

  nixpkgs.config.allowUnfree = true;

  # Load nvidia driver for Xorg and Wayland
  services.xserver.videoDrivers = ["nvidia"];

  hardware.nvidia = {
    modesetting.enable = true;
    powerManagement.enable = false;
    powerManagement.finegrained = false;
    open = false;
    package = config.boot.kernelPackages.nvidiaPackages.beta;
  };

  [...]
  
  programs.nix-ld.enable = true;

  nix.settings.experimental-features = [ "nix-command" "flakes" ];
}

@Titus-von-Koeller
Copy link
Author

output of erroring command (trying to get the bitsandbytes Python package to work):

❯ pixi run healthcheck
✨ Pixi task (healthcheck in default): python -m bitsandbytes
The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
CUDA specs: None

Torch says CUDA is not available. Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
The directory listed in your path is found to be non-existent: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64
[...] # many other similar lines

Found duplicate CUDA runtime files (see below).

We select the PyTorch default CUDA runtime, which is 12.1,
but this might mismatch with the CUDA version that is needed for bitsandbytes.
To override this behavior set the `BNB_CUDA_VERSION=<version string, e.g. 122>` environmental variable.

For example, if you want to use the CUDA version 122,
    BNB_CUDA_VERSION=122 python ...

OR set the environmental variable in your .bashrc:
    export BNB_CUDA_VERSION=122

In the case of a manual override, make sure you set LD_LIBRARY_PATH, e.g.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2,
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12.2.140
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12.2.140
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12.2.140
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12
* Found CUDA runtime at: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Checking that the library is importable and CUDA is callable...
Traceback (most recent call last):
  File "/home/titus/src/bnb/bitsandbytes/diagnostics/main.py", line 66, in main
    sanity_check()
  File "/home/titus/src/bnb/bitsandbytes/diagnostics/main.py", line 33, in sanity_check
    p = torch.nn.Parameter(torch.rand(10, 10).cuda())
  File "/home/titus/src/bnb/.pixi/envs/default/lib/python3.8/site-packages/torch/cuda/__init__.py", line 293, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

@Titus-von-Koeller
Copy link
Author

❯ nix run -f '<nixpkgs>' cudaPackages.saxpy
trace: warning: cudaPackages.autoAddDriverRunpath is deprecated, use pkgs.autoAddDriverRunpath instead
Start
Runtime version: 12020
Driver version: 0
Host memory initialized, copying to the device
CUDA error at cudaMalloc(&xDevice, N * sizeof(float)): CUDA driver is a stub library

@Titus-von-Koeller
Copy link
Author

❯ nvidia-smi
Fri Jun 28 17:09:48 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.78                 Driver Version: 550.78         CUDA Version: ERR!     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0  On |                  Off |
|  0%   47C    P3             63W /  480W |    4607MiB /  24564MiB |     39%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:03:00.0 Off |                  Off |
|  0%   37C    P8             27W /  480W |      12MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

@Titus-von-Koeller
Copy link
Author

❯ ls /run/opengl-driver/lib/libcuda*
 /run/opengl-driver/lib/libcuda.so -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcuda.so*
 /run/opengl-driver/lib/libcuda.so.1 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcuda.so.1*
 /run/opengl-driver/lib/libcuda.so.550.78 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcuda.so.550.78*
 /run/opengl-driver/lib/libcudadebugger.so -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcudadebugger.so*
 /run/opengl-driver/lib/libcudadebugger.so.1 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcudadebugger.so.1*
 /run/opengl-driver/lib/libcudadebugger.so.550.78 -> /nix/store/2paf3i2g4arx5j4m9l87zdrzsikwmizh-nvidia-x11-550.78-6.6.35/lib/libcudadebugger.so.550.78*

@Titus-von-Koeller
Copy link
Author

Titus-von-Koeller commented Jun 28, 2024

❯ LD_DEBUG=libs saxpy
     31778:     find library=libcublas.so.12 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs            (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v4/libcublas.so.12
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v3/libcublas.so.12
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/glibc-hwcaps/x86-64-v2/libcublas.so.12
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublas.so.12
     31778:
     31778:     find library=libcudart.so.12 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12
     31778:
     31778:     find library=libcublasLt.so.12 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublasLt.so.12
     31778:
     31778:     find library=libdl.so.2 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2:/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib64/stubs/libdl.so.2
     31778:      search path=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v4:/run/opengl-driver/lib/glibc-hwcaps/x86-64-v3:/run/opengl-driver/lib/glibc-hwcaps/x86-64-v2:/run/opengl-driver/lib          (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/run/opengl-driver/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/run/opengl-driver/lib/libdl.so.2
     31778:      search path=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v4:/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v3:/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v2:/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v4:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v3:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v2:/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/qdsimqhpg5ln8fy3xnaywrqdqsp7sglh-glib-2.80.2-bin/lib/libdl.so.2
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libdl.so.2
     31778:      search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v4:/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v3:/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v2:/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib          (RUNPATH from file saxpy)
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libdl.so.2
     31778:      search path=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v4:/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v3:/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v2:/run/opengl-driver-32/lib              (LD_LIBRARY_PATH)
     31778:       trying file=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/run/opengl-driver-32/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/run/opengl-driver-32/lib/libdl.so.2
     31778:      search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v4:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v3:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v2:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v4:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v3:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v2:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib               (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libdl.so.2
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libdl.so.2
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libdl.so.2
     31778:      search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v4:/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v3:/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v2:/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib              (system search path)
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v4/libdl.so.2
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libdl.so.2
     31778:
     31778:     find library=librt.so.1 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/librt.so.1
     31778:      search path=/run/opengl-driver/lib             (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/librt.so.1
     31778:      search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/librt.so.1
     31778:      search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib             (RUNPATH from file saxpy)
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/librt.so.1
     31778:      search path=/run/opengl-driver-32/lib          (LD_LIBRARY_PATH)
     31778:       trying file=/run/opengl-driver-32/lib/librt.so.1
     31778:      search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib            (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/librt.so.1
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/librt.so.1
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/librt.so.1
     31778:      search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib              (system search path)
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/librt.so.1
     31778:
     31778:     find library=libpthread.so.0 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libpthread.so.0
     31778:      search path=/run/opengl-driver/lib             (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libpthread.so.0
     31778:      search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libpthread.so.0
     31778:      search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib             (RUNPATH from file saxpy)
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libpthread.so.0
     31778:      search path=/run/opengl-driver-32/lib          (LD_LIBRARY_PATH)
     31778:       trying file=/run/opengl-driver-32/lib/libpthread.so.0
     31778:      search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib            (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libpthread.so.0
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libpthread.so.0
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libpthread.so.0
     31778:      search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib              (system search path)
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libpthread.so.0
     31778:
     31778:     find library=libstdc++.so.6 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libstdc++.so.6
     31778:      search path=/run/opengl-driver/lib             (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libstdc++.so.6
     31778:      search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libstdc++.so.6
     31778:      search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib             (RUNPATH from file saxpy)
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
     31778:
     31778:     find library=libm.so.6 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libm.so.6
     31778:      search path=/run/opengl-driver/lib             (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libm.so.6
     31778:      search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libm.so.6
     31778:      search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib             (RUNPATH from file saxpy)
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libm.so.6
     31778:      search path=/run/opengl-driver-32/lib          (LD_LIBRARY_PATH)
     31778:       trying file=/run/opengl-driver-32/lib/libm.so.6
     31778:      search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib            (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libm.so.6
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libm.so.6
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libm.so.6
     31778:      search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib              (system search path)
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libm.so.6
     31778:
     31778:     find library=libgcc_s.so.1 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libgcc_s.so.1
     31778:      search path=/run/opengl-driver/lib             (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libgcc_s.so.1
     31778:      search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libgcc_s.so.1
     31778:      search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib             (RUNPATH from file saxpy)
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libgcc_s.so.1
     31778:
     31778:     find library=libc.so.6 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libc.so.6
     31778:      search path=/run/opengl-driver/lib             (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libc.so.6
     31778:      search path=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib             (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/svh3pgaaisqsnv7wk5i5xf270a5n7p2g-libglvnd-1.7.0/lib/libc.so.6
     31778:      search path=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib             (RUNPATH from file saxpy)
     31778:       trying file=/nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libc.so.6
     31778:      search path=/run/opengl-driver-32/lib          (LD_LIBRARY_PATH)
     31778:       trying file=/run/opengl-driver-32/lib/libc.so.6
     31778:      search path=/run/opengl-driver/lib:/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib:/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib            (RUNPATH from file saxpy)
     31778:       trying file=/run/opengl-driver/lib/libc.so.6
     31778:       trying file=/nix/store/ldq93dc0pvdj9fskqrkh6zsr45jh7sx7-libcublas-12.2.5.6/lib/libc.so.6
     31778:       trying file=/nix/store/mdqwq2v8svja3rmk2xb9rq3ikx44vdfd-cuda_cudart-12.2.140/lib/libc.so.6
     31778:      search path=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib              (system search path)
     31778:       trying file=/nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
     31778:
     31778:
     31778:     calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/ld-linux-x86-64.so.2
     31778:
     31778:
     31778:     calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
     31778:
     31778:
     31778:     calling init: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libgcc_s.so.1
     31778:
     31778:
     31778:     calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libm.so.6
     31778:
     31778:
     31778:     calling init: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6
     31778:
     31778:
     31778:     calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libpthread.so.0
     31778:
     31778:
     31778:     calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/librt.so.1
     31778:
     31778:
     31778:     calling init: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libdl.so.2
     31778:
     31778:
     31778:     calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublasLt.so.12
     31778:
     31778:     find library=libcuda.so.1 [0]; searching
     31778:      search path=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib           (LD_LIBRARY_PATH)
     31778:       trying file=/nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcuda.so.1
     31778:
     31778:
     31778:     calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcuda.so.1
     31778:
     31778:
     31778:     calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12
     31778:
     31778:
     31778:     calling init: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublas.so.12
     31778:
     31778:
     31778:     initialize program: saxpy
     31778:
     31778:
     31778:     transferring control: saxpy
     31778:
Start
Runtime version: 12020
Driver version: 0
Host memory initialized, copying to the device
CUDA error at cudaMalloc(&xDevice, N * sizeof(float)): CUDA driver is a stub library
     31778:
     31778:     calling fini:  [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublas.so.12 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcudart.so.12 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcublasLt.so.12 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libdl.so.2 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/librt.so.1 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libpthread.so.0 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libstdc++.so.6 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libm.so.6 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/p44qan69linp3ii0xrviypsw2j4qdcp2-gcc-13.2.0-lib/lib/libgcc_s.so.1 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/ld-linux-x86-64.so.2 [0]
     31778:
     31778:
     31778:     calling fini: /nix/store/7w8bfjj6njzc8gbyqibq3ss4yp4fkjky-cuda-merged-12.2/lib/libcuda.so.1 [0]
     31778:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment