FRidh/overview.md

## overview.md

      
    Raw
  

              overview.md
            
          
    Python on Nix infrastructure

Introduction

Python is used throughout Nixpkgs. We use it for certain scripts, we provide
Python libraries, and we provide applications. There are several methods on how to use
Python on Nix, each with their pros and cons.
An overview of all issues with the current Python infrastructure on Nix is available in the placeholder issue 1819.
Goal

While many things work really nice there definitely still are
issues. This document states how we intend to support Python on Nix, it
describes our current infrastructure, and contains a proposal for an improved
infrastructure that supports the following use cases:

installing Python applications in a profile. These should expose the program but not the Python modules.
creating environments for Python development like virtualenv but with the additional possibility of including other non-Python programs.
temporary Python environments but also a permanent environment by installing it in a profile.
Python programs that call other Python programs without mangling the search path for modules. That means e.g. that a Python 2 program can call a Python 3 program without issues.
namespace packages
combine any of the above without issues.

Furthermore, we would like to support the following Python tools:

virtualenv for creating virtual environments. While nix-shell can do the same and more many still need virtualenv.
tox for testing against multiple environments.
nuitka, a Python compiler that depends on SCons. The challenge here is that SCons is a Python 2.7 tool while Nuitka can work with any CPython version.

Specific test cases

The following are test cases for each of the issues:
TODO
Issues solved

The following issues are supposed to be solved

#11423: Have a Python package on PATH without adding it to PYTHONPATH
#16591: PYTHONPATH leaks in subprocesses.
#22688: Do not use --prefix PYTHONPATH because it leaks PYTHONPATH.
#23676: Subprocesses do not have modules on their sys.path.
#24128: wrapPythonPrograms should not add (propagated)BuildInputs build inputs to wrappers.

Overview of Python packaging

Applications, libraries and environments

A common distinction to make is that between applications and libraries.
An application is a standalone program. The application can depend on Python
libraries but any libraries provided by the application (direct or indirect)
shouldn't be shared or integrated in other applications or environments.
When developing one is interested in the interpreter, (Python) libraries
and possibly some tools that may depend on the exact environment they're used in.
An environment provides all the programs needed. In the case of an application
this typically means the only entry point provided by the environment is the
application itself, whereas in the case of development environments multiple
tools may be available.
Let's clarify each with an example. The e-book suite Calibre is a program that
is written in Python. When using Calibre one is not interested in any libraries.
One just wants to use the program and thus we call this an application.
The package numpy is a library, and is used for development. It does provide
the program f2py2 but this is typically only used in conjunction with the
development numpy is used in. A similar example is pytest; one typically
uses pytest in the actual development environment.
In some cases this distinction may not be so clear. E.g. the Jupyter Notebook
is an application that is used for development. It depends on a kernel which is chosen
for the environment one uses for development, e.g. a Python 2 or 3 kernel.
However, it supports multiple kernels simultaneously so one could separate the
package, having the Notebook as an application and the kernels as libraries.
Distributing and installing Python code

Python code can be distributed in different ways.
The most common format is a Source Distribution
or "sdist". This contains the essential source code along with some meta
data for pip. Source distributions are typically installed with setuptools
and can be recognized by its setup.py file. They used to be installed with
python setup.py install but are nowadays commonly installed in two steps by first
creating a wheel
with python setup.py bdist_wheel and then installing the wheel.
The wheel isn't just an intermediate step in the building process but is
also a popular distribution format. A wheel is a
Built Distribution.
Wheels are often pure Python but can contain binary code. Wheels are installed
with pip using pip install *.whl. While setuptools is most commonly used for
building wheels, there exist other tools for building wheels.
One example is flit.
In some cases installation is done entirely different, e.g. with the help of a Makefile.
Libraries can sometimes also provide Python bindings.
Finally, when developing one might want to use an editable or
development mode installation with pip install -e.
Finding Python libraries

Python modules are installed in lib/pythonX.X/site-packages/<pname>. Installed
right next to it is the dist-info folder,
lib/pythonX.X/site-packages/<pname>-<version>.dist-info.
This folder is needed for pip/setuptools to determine which packages have
been installed.
The exact Python import logic is quite extensive. What follows is a very brief summary:

Python modules can be imported from folders that are on sys.path.
During startup of the interpreter it looks for the folder sitecustomize.py on sys.path. This file can be used to add additional site-packages folders to sys.path.
After startup, it checks the environment variable PYTHONPATH which is a list of folders. These folders are added to sys.path before everything else.
Entries that are added directly to sys.path are not recursed into. One can instead use site.addsitedir to add folders to sys.path. site.addsitedir does recurse by e.g. following .pth files.
.pth files list folders or other .pth files that can be added to sys.path.

The first entry in sys.path is special and is the directory containing the
script that was used to invoke the interpreter.
Another environment variable of interested is PYTHONHOME.
This environment variable can be used to change the location of the standard
Python libraries. By default, the libraries are searched in
prefix/lib/pythonversion and exec_prefix/lib/pythonversion, where prefix
and exec_prefix are installation-dependent directories, both defaulting to
/usr/local.
Name of and path to the program

The sys.argv attribute
represents the list of arguments passed to a Python program.
The first value, argv[0], is the script name. Its OS-dependent whether
this is a full path or not but on Linux and Darwin systems it is. If the command
was executed using the -c command line option to the interpreter, argv[0] is set
to the string '-c'. If no script name was passed to the Python interpreter,
argv[0] is an empty string.
The name and full path to the program are of interest because programs might
want to call themselves.


Current implementation of Python on Nix

Applications, libraries and environments

Python applications are spreadout throughout the Nixpkgs tree following the
general guidelines.
The file pkgs/top-level/python-packages.nix contains or refers to all Python
library expressions, and these packages can be accessed through
pkgs.pythonXX.pkgs.<name>. Typically one creates a environment with
pythonXX.withPackages or pythonXX.buildEnv.
Packaging Python packages

The main function for packaging Python packages is buildPythonPackage.
Furthermore, buildPythonApplication exists for applications. The only
difference is that buildPythonPackage modifies the name to include the
interpreter version.
An important argument is format which is used to choose between setuptools
(sdist), flit, wheel and other. The most common format is setuptools.
Wheels are also increasingly used in Nixpkgs. The last option is used when none
of the others apply. In this case the packager needs to provide a buildPhase
and installPhase.
The goal of the buildPythonPackage (and buildPythonApplication) is to
guarantee that applications work and modules can be found.
Building a package

The Python interpreter provides a setup hook that recurses into the
propagatedBuildInputs and adds the site-packages folder of each to the
environment variable PYTHONPATH. This allows the package that is being build
to find its dependencies. The hook is also run by nix-shell. While that makes
sense when building/debugging the build, it is also abused for creating
temporary environments with nix-shell -p python3.numpy python3.pytest.
The wrapPythonPrograms shell function wraps all executables in a derivation and does two things:

it uses site.addsitedir to update sys.path with dependencies. It recursively traverses propagatedBuildInputs and pythonPath.
it fixes the name, sys.argv[0], of the script. This has to be done because the wrapper moves the original script.

The buildPythonPackage function patches the shebangs of all scripts provided.
That way, the scripts can find the correct Python interpreter. It also exectutes
wrapPythonPrograms. Python applications that are installed can now find its
dependencies and will function.
Building an environment

The python.buildEnv function creates an environment that consists of symbolic
links to all files that are provided by the packages that are to be included in
the environment. The shebangs of the scripts have already been patched by
buildPythonPackage to point to the correct Python interpreter. However, that
store entry contains just the interpreter, and not other Python packages that
are to be included. Therefore, python.buildEnv not only creates symbolic links
but also wraps each script with a wrapper that sets PYTHONHOME to the
interpreter in the newly created environment. This environment can now be
installed. The python.withPackages function provides as simpler interface to
python.buildEnv.
Suggested improvements

Finding libraries


add a sitepackages.py to the interpreter that listens to NIX_PYTHONPATH and/or NIX_PYTHON_PTH with the latter referring to a .pth file.

Installing packages


when installing individual libraries in a profile, the interpreter cannot find them. It was suggested to use propagatedUserEnvPkgs.

Setting the name


use exec -a name program to set the name of program. Python however does not support exec -a.
wrap the interpreter and set the name of the program through sitecustomize.py. The attribute sys.argv is unavailable at that point.
patch the interpreter to listen to an environment variable, NIX_PYTHON_NAME, that defines the name. https://github.com/python/cpython/blob/3.5/Python/sysmodule.c#L2050

Wrap without leaking


use --set PYTHONPATH. This breaks a Python feature. See the discussion.

Proposed implementation


add a sitecustomize.py to the interpreter that listens to NIX_PYTHONPATH and/or NIX_PYTHON_PTH with the latter referring to a .pth file.
patch the interpreter to listen to an environment variable, NIX_PYTHON_NAME, that defines the name. https://github.com/python/cpython/blob/3.5/Python/sysmodule.c#L2050