Skip to content

Instantly share code, notes, and snippets.

@pombredanne
Last active September 22, 2021 14:09
Show Gist options
  • Save pombredanne/24dc375dfec8c97a732e5ec97a23f18d to your computer and use it in GitHub Desktop.
Save pombredanne/24dc375dfec8c97a732e5ec97a23f18d to your computer and use it in GitHub Desktop.
Resolving Python deps

Context

For the collection of the set of dependent packages from a given Python application, the package management tool (commonly pip) will first collect the direct dependencies and then query a PyPI APIs to collect the dependencies of each dependency.

For each dependency, there can be a version "requirement" that can be a simple exact version, a version expression (aka. version specifier) as defined in https://www.python.org/dev/peps/pep-0440/ and additional tags and constraints as specified in https://www.python.org/dev/peps/pep-0508/

In particular the required Python version of a package (or version specifier) can be set for the whole package (with the python_requires attribute) or as a marker for a given dependency, either direct or indirect.

Problem

Because of these Python version requirements and markers, it can be difficult to correctly resolve the set of dependencies of a given Python package; in practice the dependencies may differ depending on which Python version is used to run the package management tool; this may be true even if no specific Python version constraint is specified directly in an application as these may exist down in the dependency tree. Furthermore, when Python version is used to run the package management tool does not match the Python version constraints that may exist in the dependency tree, depdency resolution will fail and the package tool (e.g. pip) will therefore fail to install packages.

Solution

One approach to resolve this issue is to attempt to resolve the dependencies using all the current Python versions (e.g. from 3.6 to 3.10) and stop when one resolution completed. This can work in practices but results in a fairly complex setup typically using one or more Docker images to keep one of each of the supported Python versions.

Another approach which is proposed here would be to use a single Python version for resolution and not invoke directly pip as a subprocess, but instead call the underlying version resolution libraries and functions to reperform the resolution in a well controlled way (eventually requesting resolution for a Python version that is not the current Python version). The benefits of such an approach would be to run on a single Python version and to possibly resolve versions for specified arguments for a given alternative Python version, operating system and architecture.

@sschuberth
Copy link

The draft, and esp. the last paragraph of the proposed solution, looks fine to me and seems to match what we had discussed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment