EDIT: check out fix-python instead, make Python run "as usual" on NixOS!
- You are using NixOS
- You start working on a Python project
- You manage the dependencies in a classic Python virtual environment using pip or poetry
You won't be able to import some of those libraries.
$ nix run nixpkgs#python3 -- -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install numpy
(.venv) $ python -c 'import numpy'
You'll get for instance:
libz.so.1: cannot open shared object file: No such file or directory
When we peak at the installation of numpy
, there are a bunch of compiled files (executables or libraries).
Some of those are not properly linked.
$ nix shell nixpkgs#file --command find .venv/lib/python3.9/site-packages/numpy -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print -exec sh -c "ldd '{}' | grep 'not found'" \;
.venv/lib/python3.9/site-packages/numpy/fft/_pocketfft_internal.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/bit_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_common.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_pcg64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_mt19937.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_bounded_integers.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_philox.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/mtrand.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_sfc64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/linalg/_umath_linalg.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/core/_rational_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_operand_flag_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_umath_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_struct_ufunc_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_simd.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
In the example above, we can see that lapack_lite.cpython-39-x86_64-linux-gnu.so
, one of the binary files provided by the numpy
wheel, does not find the libz.so.1
library.
-
Install the dependencies, or enter a shell
$ nix shell nixpkgs#file nixpkgs#patchelf github:GuillaumeDesforges/nix-patchtools
-
Set the environment variable
libs
(used byautopatchelf
) to a list of paths to all the required libraries.$ export libs=$(nix eval --expr 'let pkgs = import (builtins.getFlake "nixpkgs") {}; in pkgs.lib.strings.makeLibraryPath (with pkgs; [ gcc.cc zlib glibc ])' --impure | sed 's/^"\(.*\)"$/\1/'):$(find $(pwd) -name '*.libs' | tr '\n' ':' | sed 's/:$//')
-
Save the following script to a file and run it
TMP="__tmp" mkdir -p $TMP cd $TMP find "$(realpath "../.venv")" -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print \ | while read file do echo "file=$file" RPATH=$(patchelf --print-rpath "$file") echo "rpath=$RPATH" autopatchelf "$file" if [[ "$RPATH" =~ .*(^|:)[^/].* ]]; then echo "repatch rpath" patchelf --set-rpath "" "$file" autopatchelf "$file" fi done cd .. rm -R $TMP
After running the script, check the result:
$ . .venv/bin/activate
(.venv) $ python -c 'import numpy'
@romain-keramitas-prl heya! Thanks for sharing this, indeed LD_LIBRARY_PATH is one option, so are Nox-specific linkers like
nix-ld
.A downside of LD_LIBRARY_PATH is that you need to be able to set said environment variable for the process that will start Python. So if it's in a bash it's fairly simple. In some other cases, that can start to be a pain, e.g. when using VSCode and its extensions that can spawn Python processes. So fixing the binaries once and for all looks more robust to me. It med to less headaches in my experience.
I should add some alternatives to this gist, but honestly this is just a rough dump of some ideas.