EDIT: check out fix-python instead, make Python run "as usual" on NixOS!
- You are using NixOS
- You start working on a Python project
- You manage the dependencies in a classic Python virtual environment using pip or poetry
You won't be able to import some of those libraries.
$ nix run nixpkgs#python3 -- -m venv .venv
$ . .venv/bin/activate
(.venv) $ pip install numpy
(.venv) $ python -c 'import numpy'
You'll get for instance:
libz.so.1: cannot open shared object file: No such file or directory
When we peak at the installation of numpy
, there are a bunch of compiled files (executables or libraries).
Some of those are not properly linked.
$ nix shell nixpkgs#file --command find .venv/lib/python3.9/site-packages/numpy -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print -exec sh -c "ldd '{}' | grep 'not found'" \;
.venv/lib/python3.9/site-packages/numpy/fft/_pocketfft_internal.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/bit_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_common.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_pcg64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_mt19937.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_bounded_integers.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_philox.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/mtrand.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_sfc64.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/random/_generator.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/linalg/_umath_linalg.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/linalg/lapack_lite.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
.venv/lib/python3.9/site-packages/numpy/core/_rational_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_operand_flag_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_umath_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_struct_ufunc_tests.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_simd.cpython-39-x86_64-linux-gnu.so
.venv/lib/python3.9/site-packages/numpy/core/_multiarray_umath.cpython-39-x86_64-linux-gnu.so
libz.so.1 => not found
In the example above, we can see that lapack_lite.cpython-39-x86_64-linux-gnu.so
, one of the binary files provided by the numpy
wheel, does not find the libz.so.1
library.
-
Install the dependencies, or enter a shell
$ nix shell nixpkgs#file nixpkgs#patchelf github:GuillaumeDesforges/nix-patchtools
-
Set the environment variable
libs
(used byautopatchelf
) to a list of paths to all the required libraries.$ export libs=$(nix eval --expr 'let pkgs = import (builtins.getFlake "nixpkgs") {}; in pkgs.lib.strings.makeLibraryPath (with pkgs; [ gcc.cc zlib glibc ])' --impure | sed 's/^"\(.*\)"$/\1/'):$(find $(pwd) -name '*.libs' | tr '\n' ':' | sed 's/:$//')
-
Save the following script to a file and run it
TMP="__tmp" mkdir -p $TMP cd $TMP find "$(realpath "../.venv")" -type f -executable -exec sh -c "file -i '{}' | grep -qE 'x-(.*); charset=binary'" \; -print \ | while read file do echo "file=$file" RPATH=$(patchelf --print-rpath "$file") echo "rpath=$RPATH" autopatchelf "$file" if [[ "$RPATH" =~ .*(^|:)[^/].* ]]; then echo "repatch rpath" patchelf --set-rpath "" "$file" autopatchelf "$file" fi done cd .. rm -R $TMP
After running the script, check the result:
$ . .venv/bin/activate
(.venv) $ python -c 'import numpy'
Thanks for the heads up @Melkor333 ! Note that I have wrapped my idea from here into a more usable tool: https://github.com/GuillaumeDesforges/fix-python/