Contemporary Python Packaging
This document lays out a set of Python packaging practices. I don't claim they are best practices, but they fit my needs, and might fit yours.
This document has been superseded as of July 2020.
This was written in July 2019. As of this writing Python 2.7 and Python 3.5 still have not reached end-of-life. This document should be superseded or disregarded no later than the Python 3.6 end-of-life. If you cite this as a justification for your behavior, please stop doing so at that time.
- For versioning, use versioneer
- For describing your build requirements, use
- For all static (and some dynamic-ish) metadata, use
- A very small
setup.pyfor dynamic metadata and to tie everything together
- Go ahead and keep
My position in the Python ecosystem will color my perspective and approach to packaging, and, presumably, how much weight you give what I think.
I am most active with the NIPY collection of projects, generally related to neuroimaging and neuroscience. I am currently the lead maintainer of NiBabel and do a reasonable amount of maintenance work for Nipype, PyBIDS, fMRIPrep and a few other packages closely related to the aforementioned.
I am not an active developer in CPython, PyPA or any packaging-related tools. I have not followed deep arguments, but have relied mostly on PEPs, documentation and sporadic searches to identify the current state of the art.
So my perspective is less concerned with what packaging should become, and more with what works today and where things appear to be heading, as I look to prepare or update packaging infrastructure for several tools. Infrastructure I hope to stop thinking about so much.
Motivating my recommendations are a few desiderata, in rough order of importance:
- Installation should work, from source, on fairly old systems. Debian Stable (9) is my touchstone here.
- Prefer declarative syntax, and limit dynamic metadata, as much as possible.
- Enable revision-based versions, with minimal opportunity for error.
- Limit custom code to absolute minimum. (Partially redundant with limiting dynamic metadata.)
To operationalize (1), the following approaches should all install correctly:
python setup.py install
pip install .
python setup.py sdist && pip install dist/*.tar.gz
python setup.py bdist_wheel && pip install dist/*.whl
And development/editable mode should work:
python setup.py develop
pip install -e .
This also means that newer, better build systems that do not rely on setuptools are not really under consideration here.
To operationalize (3), all of the above should produce an install with the same version
string, and setting the version should be done from a version control tag if possible.
git repository, the following should also work:
git archive -o archive.tar.gz $TAG && pip install archive.tar.gz
A bare minimum
pyproject.toml is as follows:
[build-system] requires = ["setuptools >= 30.3.0", "wheel"]
Additional build dependencies such as
numpy might be put here.
As of setuptools 30.3.0, most packaging metadata can be set declaratively in
The following skeleton can be used as a model.
[metadata] url = https://github.com/your/package author = You author_email = firstname.lastname@example.org maintainer = You maintainer_email = email@example.com description = A package long_description = file:README.rst long_description_content_type = text/x-rst; charset=UTF-8 license = GPL classifiers = Programming Language :: Python [options] python_requires = >= 3.5 install_requires = test_requires = pytest coverage packages = find: include_package_data = True [options.package_data] * = data/* [options.extras_require] doc = sphinx test = pytest coverage all = %(doc)s %(test)s
I want to draw attention to the
python_requires metadata which will prevent
attempting to install on incompatible systems. When you drop 2.7 and 3.4 - or any other
versions - update the
python_requires to avoid breaking downstream tools that still
support those versions.
In addition to plain key-value pairs, there are some constrained options for common
dynamic metadata. For example,
long_description = file:<filename> allows you to place
a long description in a separate file, to be included in your documentation.
packages = find: replaces the
find_packages() option often used in
Finally, interpolated strings are used in
extras_require to provide a meta-extra like
I recommend not placing the version in
The dynamic components of my package setup are as follows:
#!/usr/bin/env python import sys from setuptools import setup import versioneer # Give setuptools a hint to complain if it's too old a version # 30.3.0 allows us to put most metadata in setup.cfg # Should match pyproject.toml SETUP_REQUIRES = ['setuptools >= 30.3.0'] # This enables setuptools to install wheel on-the-fly SETUP_REQUIRES += ['wheel'] if 'bdist_wheel' in sys.argv else  if __name__ == '__main__': setup(name='package', version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), setup_requires=SETUP_REQUIRES, )
I place the package name in
setup.py mostly because, without this, GitHub will not recognize
your package to place it in its dependency graphs.
setup.py as opposed to adding
version = attr:package.__version__ to
setup.cfg, we avoid the issue of missing import-time dependencies.
setuptools how to encode the current version into various
setup_requires is mostly here as a fall-back to let old versions of setuptools
provide a user-readable explanation for failures.
Versioneer will set the version based on your git tag, and handle all of the install cases I described in desiderata.
This requires an additional section to your
[versioneer] VCS = git style = pep440 versionfile_source = package/_version.py versionfile_build = package/_version.py tag_prefix = parentdir_prefix =
It can then be installed from your repository root with:
pip install versioneer versioneer install
Once done, it places a copy of itself in your repository root, so other users do not need to install it for it to be used correctly.
N.B. Versioneer does not work out of the box with git archives for non-tag releases.
If you need any archived revision, this will not be sufficient. I don't know of a general
solution to that problem at this point, as
git archive substitution is quite limited.
I had hoped to be able to eliminate
but source distributions (
sdists) do not seem able to package correctly without it.
It is almost certainly possible to write a helper function in
setup.py that will populate it from
setup.cfg metadata, but a new batch of custom code feels somewhat counter to the spirit of
I've elected to simply wait for a couple more years and re-assess.
Notes on new build systems and legacy operating systems
As noted above, setuptools was the only system under serious consideration, simply because it has
long been the standard to run
python setup.py. Until pip 10+ is universal, alternative build
systems will create headaches that I don't want to deal with.
CentOS 7, for instance still packages pip 7.1 and setuptools 0.9.8, which means the above will not
work out of the box. However, sticking with setuptools and
setup_requires ensures that a user
will at least be told to upgrade setuptools.
To the extent copyright can be claimed, I disclaim it under CC0.