This document lays out a set of Python packaging practices. I don't claim they are best practices, but they fit my needs, and might fit yours.
This document has been superseded as of July 2020.
This was written in July 2019. As of this writing Python 2.7 and Python 3.5 still have not reached end-of-life. This document should be superseded or disregarded no later than the Python 3.6 end-of-life. If you cite this as a justification for your behavior, please stop doing so at that time.
- For versioning, use versioneer
- For describing your build requirements, use
pyproject.toml - For all static (and some dynamic-ish) metadata, use
setup.cfg - A very small
setup.pyfor dynamic metadata and to tie everything together - Go ahead and keep
MANIFEST.infor now
My position in the Python ecosystem will color my perspective and approach to packaging, and, presumably, how much weight you give what I think.
I am most active with the NIPY collection of projects, generally related to neuroimaging and neuroscience. I am currently the lead maintainer of NiBabel and do a reasonable amount of maintenance work for Nipype, PyBIDS, fMRIPrep and a few other packages closely related to the aforementioned.
I am not an active developer in CPython, PyPA or any packaging-related tools. I have not followed deep arguments, but have relied mostly on PEPs, documentation and sporadic searches to identify the current state of the art.
So my perspective is less concerned with what packaging should become, and more with what works today and where things appear to be heading, as I look to prepare or update packaging infrastructure for several tools. Infrastructure I hope to stop thinking about so much.
Motivating my recommendations are a few desiderata, in rough order of importance:
- Installation should work, from source, on fairly old systems. Debian Stable (9) is my touchstone here.
- Prefer declarative syntax, and limit dynamic metadata, as much as possible.
- Enable revision-based versions, with minimal opportunity for error.
- Limit custom code to absolute minimum. (Partially redundant with limiting dynamic metadata.)
To operationalize (1), the following approaches should all install correctly:
python setup.py installpip install .python setup.py sdist && pip install dist/*.tar.gzpython setup.py bdist_wheel && pip install dist/*.whl
And development/editable mode should work:
python setup.py developpip install -e .
This also means that newer, better build systems that do not rely on setuptools are not really under consideration here.
To operationalize (3), all of the above should produce an install with the same version
string, and setting the version should be done from a version control tag if possible.
Assuming a git repository, the following should also work:
git archive -o archive.tar.gz $TAG && pip install archive.tar.gz
I have settled on a setuptools-based approach,
using setup.cfg to declare as much of the metadata as possible, along with pyproject.toml
laid out in PEP 518.
Versioneer is used to handle versioning.
A bare minimum pyproject.toml is as follows:
[build-system]
requires = ["setuptools >= 30.3.0", "wheel"]Additional build dependencies such as cython and numpy might be put here.
As of setuptools 30.3.0, most packaging metadata can be set declaratively in
setup.cfg.
The following skeleton can be used as a model.
[metadata]
url = https://github.com/your/package
author = You
author_email = your@email.tld
maintainer = You
maintainer_email = your@email.tld
description = A package
long_description = file:README.rst
long_description_content_type = text/x-rst; charset=UTF-8
license = GPL
classifiers =
Programming Language :: Python
[options]
python_requires = >= 3.5
install_requires =
test_requires =
pytest
coverage
packages = find:
include_package_data = True
[options.package_data]
* =
data/*
[options.extras_require]
doc =
sphinx
test =
pytest
coverage
all =
%(doc)s
%(test)sI want to draw attention to the python_requires metadata which will prevent pip from
attempting to install on incompatible systems. When you drop 2.7 and 3.4 - or any other
versions - update the python_requires to avoid breaking downstream tools that still
support those versions.
In addition to plain key-value pairs, there are some constrained options for common
dynamic metadata. For example, long_description = file:<filename> allows you to place
a long description in a separate file, to be included in your documentation.
packages = find: replaces the find_packages() option often used in setup.py.
Finally, interpolated strings are used in extras_require to provide a meta-extra like all.
I recommend not placing the version in setup.cfg.
The dynamic components of my package setup are as follows:
#!/usr/bin/env python
import sys
from setuptools import setup
import versioneer
# Give setuptools a hint to complain if it's too old a version
# 30.3.0 allows us to put most metadata in setup.cfg
# Should match pyproject.toml
SETUP_REQUIRES = ['setuptools >= 30.3.0']
# This enables setuptools to install wheel on-the-fly
SETUP_REQUIRES += ['wheel'] if 'bdist_wheel' in sys.argv else []
if __name__ == '__main__':
setup(name='package',
version=versioneer.get_version(),
cmdclass=versioneer.get_cmdclass(),
setup_requires=SETUP_REQUIRES,
)I place the package name in setup.py mostly because, without this, GitHub will not recognize
your package to place it in its dependency graphs.
By using versioneer in setup.py as opposed to adding version = attr:package.__version__ to
the setup.cfg, we avoid the issue of missing import-time dependencies.
versioneer.get_cmdclass() tells setuptools how to encode the current version into various
installation methods.
Finally, setup_requires is mostly here as a fall-back to let old versions of setuptools
provide a user-readable explanation for failures.
Versioneer will set the version based on your git tag, and handle all of the install cases I described in desiderata.
This requires an additional section to your setup.cfg:
[versioneer]
VCS = git
style = pep440
versionfile_source = package/_version.py
versionfile_build = package/_version.py
tag_prefix =
parentdir_prefix = It can then be installed from your repository root with:
pip install versioneer
versioneer installOnce done, it places a copy of itself in your repository root, so other users do not need to install it for it to be used correctly.
N.B. Versioneer does not work out of the box with git archives for non-tag releases.
If you need any archived revision, this will not be sufficient. I don't know of a general
solution to that problem at this point, as git archive substitution is quite limited.
I had hoped to be able to eliminate
MANIFEST.in,
but source distributions (sdists) do not seem able to package correctly without it.
It is almost certainly possible to write a helper function in setup.py that will populate it from
the setup.cfg metadata, but a new batch of custom code feels somewhat counter to the spirit of
this endeavor.
I've elected to simply wait for a couple more years and re-assess.
The minimum setuptools version needed for setup.cfg to work is
30.3.0, although more fields
have been defined since then, and the minimum pip needed for PEP 518 compatibility is
10.0.0.
As noted above, setuptools was the only system under serious consideration, simply because it has
long been the standard to run python setup.py. Until pip 10+ is universal, alternative build
systems will create headaches that I don't want to deal with.
CentOS 7, for instance still packages pip 7.1 and setuptools 0.9.8, which means the above will not
work out of the box. However, sticking with setuptools and setup_requires ensures that a user
will at least be told to upgrade setuptools.
To the extent copyright can be claimed, I disclaim it under CC0.