Skip to content

Instantly share code, notes, and snippets.

@rbtcollins

rbtcollins/build-system-abstraction.rst

Last active October 27, 2015 01:38
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rbtcollins/666c12aec869237f7cf7 to your computer and use it in GitHub Desktop.
Save rbtcollins/666c12aec869237f7cf7 to your computer and use it in GitHub Desktop.
Download ZIP
build system abstraction
Raw
build-system-abstraction.rst
PEP

XX

Title

Build system abstraction for pip/conda etc

Version

$Revision$

Last-Modified

$Date$

Author

Robert Collins <rbtcollins@hp.com>, Nathaniel Smith <njs@pobox.com>

Discussions-To

distutils-sig <distutils-sig@python.org>

Status

Draft

Type

Standards Track

Content-Type

text/x-rst

Created

26-Oct-2015

Post-History

XX

Requires

PEP-426

Abstract

This PEP specifies a programmatic interface for pip1 and other distribution or installation tools to use when working with Python source trees (both the developer tree - e.g. the git tree - and source distributions).

The programmatic interface allows decoupling of pip from its current hard dependency on setuptools2 able for two key reasons:

  1. It enables new boutique build systems that may be much easier to use without requiring them to even appear to be setuptools.
  2. It facilitates setuptools itself changing it's user interface without breaking pip, giving looser coupling.

The programmatic interface also enables pip to install build time requirements for packages which is an important step in getting pip to full feature parity with the instalation components of easy-install.

As written it depends on PEP-426 which is draft. Sadly PEP-345 does not define setup-requires/build-requires metadata, so we either need a narrow PEP to add that to PEP-345, or to ratify PEP-426, or to break out just-enough-from-pep-426 into a new narrow spec.

Motivation

There is significant pent-up frustration in the Python packaging ecosystem around the current lock-in between build system and pip. Breaking that lock-in is better for pip, for setuptools, and for other build systems like flit3.

Specification

Overview

Build tools will be located by reading a file pypa.yaml from the root directory of the source tree. That file describes how to get the build tool. The build tool is then self-describing. This avoids having stale descriptions of build tools encoded into sdists that have been uploaded to PyPI4.

The interface involves executing processes rather than loading Python modules, because tools like pip will often interact with many different versions of the same build tool during a single invocation, and in-process APIs are much harder to manage in that situation.

Process interface

Where a process needs to be run, we need to be able to assemble the subprocess call to make. For this we use a simple command string with in-Python variable interpolation. Only variables defined in this PEP will be honoured: the use of additional variables is an error. Additional variables can only be added with a schema version increase. Basic variable expressions such as ${PYTHON}, ${PYTHON:-python}, ${PYTHON:+PYTHON -m coverage} can be used. An implementation of this is in shellvars5.

Processes will be run with the current working directory set to the root of the source tree.

Available variables

PYTHON

The Python interpreter in use.

OUTPUT_DIR

Where to create requested output. If not set, outputting to the current working directory is appropriate.

pypa.yaml

The yaml file has the following schema. Extra keys are ignored.

schema

The version of the schema. This PEP defines version 1. Defaults to 1

bootstrap-requires

The requirements required to run the build tool itself.

This should be a list of specifiers, in plain text form.

e.g.:

bootstrap-requires:
  - requirement
  - requirement[extra]
  - requirement:python_version<"3"
  - requirement[extra]:python_version>"2"
  - requirement[extra]>2.5:python_version=="2.7"
build-tool

A command to run to query the build tool for its complete interface. The build tool should output on stdout a build tool description JSON document. Stdin may not be read from, and stderr may be handled however the calling process desires.

build tool description

The build tool description schema. Extra keys are ignored.

schema

The version of the schema. This PEP defines version 1. Defaults to 1

build-requires

Command to run to query build requirements. Build requirements are returned as a partially filled out PEP-4266 JSON metadata for the project. Additional data may be included, but the build_requires and metadata_version keys must be present.

dist-info

Command to run to generate PEP-426 JSON metadata for the project. The metadata should be output on stdout, stdin may not be consumed, and stderr handling is up to the discretion of the calling process. The build-requires for the project will be present in the Python environment when the dist-info command is run.

wheel

Command to run to build a wheel of the project. OUTPUT_DIR will be set and point to an existing directory where the wheel should be output. Stdin may not be consumed, stdout and stderr handling is at the discretion of the calling process. The build-requires for the project will be present in the Python environment when the wheel command is run.

develop

Command to do an in-place 'development' installation of the project. Stdin may not be consumed, stdout and stderr handling is at the discretion of the calling process. The build-requires for the project will be present in the Python environment when the develop command is run.

Not all build systems will be able to perform develop installs. If a build system cannot do develop installs, then this key can be omitted.

Note tht when it is omitted, commands like pip install -e foo will be unable to complete.

provided-by

The distribution name that define this build system interface. This is used to facilitate caching the build tool output to avoid requiring an extra subprocess per package as a result of this specification. Specifically, where the resolved bootstrap-requires results in the same version of the named distribution being installed, the build tool description is presumed to be identical.

Python environments and hermetic builds

This specification does not prescribe whether builds should be hermetic or not. Existing build tools like setuptools will use installed versions of build time requirements (e.g. setuptools_scm) and only install other versions on version conflicts or missing dependencies. However its likely that better consistency can be created by always isolation builds and using only the specified dependencies.

However there are nuanced problems there - such as how can users force the avoidance of a bad version of a build requirement which meets some packages dependencies. Future PEPs may tackle this problem, but it is not currently in scope - it does not affect the metadata required to coordinate between build systems and things that need to do builds.

Upgrades

Both 'pypa.yaml' and the build tool description are versioned to permit future incompatible changes. There is a sequence dependency here.

Upgrades to the schemas defined in this specification must proceed with the consumers first. To ensure that consumers should refuse to operate when 'pypa.yaml' has a schema version that they do not recognise.

Build tools listed in a 'pypa.yaml' with schema version 1 must not generate a build tool description with a version other than 1 (or absent, as the default is 1).

Thus the sequence for upgrading either of schemas in a new PEP will be:

  1. Issue new PEP defining build tool description and 'pypa.yaml' schemas. The 'pypa.yaml' schema version must change if either the 'pypa.yaml' schema has changed or the build tool description schema has changed. The build tool description schema version must change if the build tool description schema has changed.
  2. Consumers (e.g. pip) implement support for the new schema version.
  3. Build tool authors implement the new schemas, and publish updated reference 'pypa.yaml' files for their users.
  4. Package authors opt into the new schema when they are happy to introduce a dependency on the version of 'pip' (and potentially other consumers) that introduced support for the new schema version.

The same process will take place for the initial deployment of this PEP:- the propogation of the capability to use this PEP without a setuptools shim will be largely gated by the adoption rate of the first version of pip that supports it.

Static metadata in sdists

This PEP does not tackle the current inability to trust static metadata in sdists. That is a separate problem to identifying and consuming the build system that is in use in a source tree, whether it came from an sdist or not.

Backwards Compatibility

Older pips will remain unable to handle alternative build systems. This is no worse than the status quo - and individual build system projects can decide whether to include a shim setup.py or not.

All existing build systems that can product wheels and do develop installs should be able to run under this abstraction and will only need a specific adapter for them constructed and published on PyPI.

In the absence of a pypa.yaml file, tools like pip should assume a setuptools build system and use setuptools commands directly.

Network effects

Projects that adopt build systems that are not setuptools compatible - that is that they have no setup.py, or the setup.py doesn't accept commands that existing tools try to use - will not be installable by those existing tools.

Where those projects are used by other projects, this effect will cascade.

In particular, because pip does not handle setup-requires today, any project (A) that adopts a setuptools-incompatible build system and is consumed as a setup-requirement by a second project (B) which has not itself transitioned to having a pypa.yaml will make B uninstallable by any version of pip. This is because setup.py in B will trigger easy-install when 'setup.py egg_info' is run by pip, and that will try and fail to install A.

As such we recommend that tools which are currently used as setup-requires either ensure that they keep a setuptools shim or find their consumers and get them all to upgrade to the use of a pypa.yaml in advance of moving themselves. Pragmatically that is impossible, so the advice is to keep a setuptools shim indefinitely - both for projects like pbr, setuptools_scm and also projects like numpy.

setuptools shim

It would be possible to write a generic setuptools shim that looks like setup.py and under the hood uses pypa.yaml to drive the builds. This is not needed for pip to use the system, but would allow package authors to use the new features while still retaining compatibility with older pip versions.

Rationale

This PEP started with a long mailing list thread on distutils-sig7. Subsequent to that a online meeting was held to debug all the positions folk had. Minutes from that were posted to the list8.

This specification is a translation of the consensus reached there into PEP form, along with some arbitrary choices on the minor remaining questions.

The basic heuristic for the design has to been to focus on introducing an abstraction without requiring development not strictly tied to the abstraction. Where the gap is small to improvements, or the cost of using the existing interface is very high, then we've taken on having the improvement as a dependency, but otherwise defered such to future iterations.

Specifically the use of PEP-426 metadata as a dependency was chosen because there are already implementations that generate PEP-426 structured data and the use of a directory on disk is sufficiently ugly that the tradeoff seems worthwhile.

The use of 'develop' as a command is because there is no PEP specifying the interoperability of things that do what 'setuptools develop' does - so we'll need to define that before pip can take on the responsibility for doing the 'develop' step.

References

Copyright

This document has been placed in the public domain.

Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

  1. pip, the recommended installer for Python packages (http://pip.readthedocs.org/en/stable/)↩

  2. setuptools, the defacto Python package build system (https://pythonhosted.org/setuptools/)↩

  3. flit, a simple way to put packages in PyPI (http://flit.readthedocs.org/en/latest/)↩

  4. PyPI, the Python Package Index (https://pypi.python.org/)↩

  5. Shellvars, an implementation of shell variable rules for Python. (https://github.com/testing-cabal/shellvars)↩

  6. PEP-426, Python distribution metadata. (https://www.python.org/dev/peps/pep-0426/)↩

  7. The kick-off thread. (https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html)↩

  8. The minutes. (https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html)↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment