How to: make Python dependencies installed via pip work on NixOS

nixos_python_patch_venv_bins.md

EDIT: check out fix-python instead, make Python run "as usual" on NixOS!

How to: make Python dependencies installed via pip work on NixOS

The issue

• You are using NixOS
• You start working on a Python project
• You manage the dependencies in a classic Python virtual environment using pip or poetry

You won't be able to import some of those libraries.

Example

$ nix run nixpkgs#python3 -- -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install numpy
(.venv) $ python -c 'import numpy'

You'll get for instance:

libz.so.1: cannot open shared object file: No such file or directory

What happens

When we peak at the installation of numpy, there are a bunch of compiled files (executables or libraries). Some of those are not properly linked.

$ nix shell nixpkgs#file --command find .venv/lib/python3.9/site-packages/numpy -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print -exec sh -c "ldd '{}' | grep 'not found'" \;
.venv/lib/python3.9/site-packages/numpy/fft/_pocketfft_internal.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/bit_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_common.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_pcg64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_mt19937.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_bounded_integers.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_philox.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/mtrand.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_sfc64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/linalg/_umath_linalg.cpython-39-x86_64-linux-gnu.so
        libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so
        libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/core/_rational_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_operand_flag_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_umath_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_struct_ufunc_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_simd.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so
        libz.so.1 => not found

In the example above, we can see that lapack_lite.cpython-39-x86_64-linux-gnu.so, one of the binary files provided by the numpy wheel, does not find the libz.so.1 library.

How to fix

1. Install the dependencies, or enter a shell

   $ nix shell nixpkgs#file nixpkgs#patchelf github:GuillaumeDesforges/nix-patchtools
   

2. Set the environment variable libs (used by autopatchelf) to a list of paths to all the required libraries.

   $ export libs=$(nix eval --expr 'let pkgs = import (builtins.getFlake "nixpkgs") {}; in pkgs.lib.strings.makeLibraryPath (with pkgs; [ gcc.cc zlib glibc ])' --impure | sed 's/^"\(.*\)"$/\1/'):$(find $(pwd) -name '*.libs' | tr '\n' ':' | sed 's/:$//')
   

3. Save the following script to a file and run it

   TMP="__tmp"
   mkdir -p $TMP
   cd $TMP
   
   find "$(realpath "../.venv")" -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print \
     | while read file
       do
         echo "file=$file"
         RPATH=$(patchelf --print-rpath "$file")
         echo "rpath=$RPATH"
         autopatchelf "$file"
         if [[ "$RPATH" =~ .*(^|:)[^/].* ]]; then
           echo "repatch rpath"
           patchelf --set-rpath "" "$file"
           autopatchelf "$file"
         fi
       done
   
   cd ..
   rm -R $TMP

After running the script, check the result:

$ . .venv/bin/activate
(.venv) $ python -c 'import numpy'

In the special case of CUDA for pytorch:

You need to add $(readlink /run/opengl-driver)/lib to $libs.

Also, since the lib is not required, autopatchelf will not add it, so the bash script does not suffice.
What you can do instead is set the RPATH of all executables to $libs:

find "$(realpath ".venv")" -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print \
  | while read file
    do
      echo "file=$file"
      RPATH=$(patchelf --print-rpath "$file")
      echo "> rpath=$RPATH"
      patchelf --set-rpath "$libs" "$file"
      echo
    done                                                                                                                                                                                     

Hey ! Got the same problem but kept digging.
FYI, there is a simpler solution when using Nix shell for the specific issue of installing numpy. It involves:

1. Adding the zlib package to your Nix env
2. Adding it's path the LD_LIBRARY_PATH environment variable, e.g.:

A minimal shell.nix file would look like this:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  nativeBuildInputs = [
    pkgs.python310
    pkgs.poetry
    pkgs.zlib
  ];

  shellHook = ''
    export LD_LIBRARY_PATH="${pkgs.zlib}/lib"
  '';
}

Found the solution via this issue, although they had the additional problem of tryin to install scipy. It seems it's not uncommon to link libs via this, or other variables. Anyway cheers 👋

@romain-keramitas-prl heya! Thanks for sharing this, indeed LD_LIBRARY_PATH is one option, so are Nox-specific linkers like nix-ld.

A downside of LD_LIBRARY_PATH is that you need to be able to set said environment variable for the process that will start Python. So if it's in a bash it's fairly simple. In some other cases, that can start to be a pain, e.g. when using VSCode and its extensions that can spawn Python processes. So fixing the binaries once and for all looks more robust to me. It med to less headaches in my experience.

I should add some alternatives to this gist, but honestly this is just a rough dump of some ideas.

Yeah I see what you mean, but I think most users may be a bit reticent to repatch libraries themselves without fully understanding what they are doing. Also to be faire I definitely hadn't in mind the VSCode spawning process aspect of it :p I'm usually working most of the time with an editor on one side and the shell on the other, so I probably overlooked it.

In all cases really cool stuff, I'll definitely come back to this if I run into issues 👌

FYI I added a link to this gist in the NixOS wiki so that others might find it more easily.
I haven't used it, so please correct me if the description isn't correct. Just thought it might be more visible that way :)

Thanks for the heads up @Melkor333 ! Note that I have wrapped my idea from here into a more usable tool: https://github.com/GuillaumeDesforges/fix-python/

JFYI you can reliably import environment variables from a flake into VSCode via https://github.com/direnv/direnv-vscode. LD_LIBRARY_PATH approach works this way.

@fjolne 👍

Some people prefer not to toy with LD_LIBRARY_PATH as in some edge cases it's not reliable, however that should cover most cases.