Python Setuptools (Packaging Guidelines) #python

python_setuptools.md

Python Setuptools

Python setuptools package replaces the distutils package. You use this in order to allow your Python package to be redistributable. Not just for PyPi but also for source distributions or private distributions.

First you need a setup.py at your project root:

#!/usr/bin/env python

from pathlib import Path
from setuptools import setup, find_packages

root_dir = Path(__file__).parent.resolve()
root_uri = root_dir.as_uri()

# assuming you have a README.md
with open(root_dir / 'README.md', mode='r', encoding='utf-8') as f:
    long_description = f.read()

# this is just a basic example
# there are many more other options
# scripts refer to scripts that are executable on the command line
# install_requires list PyPi dependencies (with optional version constraints)
# dependency_links refer to non PyPi dependencies
# package-name is the name of the python package
# modules to be imported can be represented as subdirectories with `__init__.py`
setup(
    name='package-name',
    version='0.0.1',
    author='Your Name',
    author_email='your-email@email.com',
    description='Cool package that does something',
    long_description=long_description,
    url='https://github.com/your-name/your-package',
    packages=find_packages(),
    scripts=['some-script'],
    install_requires=['dep1', 'dep2>=1', 'dep3>=1,<2'],
    dependency_links=[
        'https://github.com/user/repo/tarball/master#egg=package-1.0',
        'git+ssh://git@github.com/account/repo.git@92364962f6b695661f35a117bf11f96584128a8d#egg=package-1.0',
        root_uri + '/subdirectory#egg=subdirectory-0.0.1'
    ])

Beware that if you are using dependency_links with SSH URIs, you need to use / between the domain and the account name. This is not the same URI you get from Github or Gitlab. Also you always need to provide a version number in a dependency link. However the install_requires does not need a version constraint.

If you're using Nix, you probably want a shell.nix to be distributed as well. In this case, you should have a MANIFEST.in that includes the shell.nix:

include README.md
include LICENSE
include requirements.txt
include shell.nix

Since I use Nix, I'm going to assume the shell.nix exists as well. The contents of the shell.nix would be a different topic.

Your project structure should look like:

.
├── MANIFEST.in
├── package-name
├── setup.py
└── shell.nix

1 directory, 3 files

Inside package-name there should be an __init__.py and you can write more modules there. Without actual directories with __init__.py, there's nothing you can actually import ... in the Python interpreter.

At this point you can try to install your dependencies specified inside your setup.py using pip install .. Any dependency already installed by Nix via shell.nix will be ignored by pip.

This will technically install your project into your Python installation PIP_PREFIX. But because your project has nothing interesting in it, there's really nothing that can be imported. However all of the dependencies specified in install_requires will be installed.

If you make changes to your code and run pip install . again, pip won't know that things have changed, so it won't actually change anything. Instead you have to remove the package via pip uninstall package-name && pip install .. I'm not sure if there's a faster way to do this at this moment.

Note that you probably want to use pip install --editable . instead, as this ensures that what you install into your PIP_PREFIX will actually automatically update to whatever changes you make to your current module. pip will utilise platform-agnostic symbolic links to make this work. Updates will be automatically propagated to all other projects that utilise the same PIP_PREFIX. These editable packages should be uninstallable as well using pip uninstall package-name. Using pip install --editable . also ensures that you can perform imports on your module no matter where the code is located.

The setup.py script is actually useful for other things as well.

./setup.py install # superseded by pip install .
./setup.py develop # superseded by pip install -e .
./setup.py build # build the package into the build directory
./setup.py sdist # produce the source distribution archive in dist directory
./setup.py register # register package against PyPi
./setup.py upload # upload the package to PyPi

You may also need a setup.cfg file, which adds extra metadata to the package.

A little note about requirements.txt. This represents a fixed set of packages that another developer can use to replicate your Python environment. To use it just make sure to run pip freeze > requirements.txt, whenever you want to update the environment. When you want to use it, just use pip install -r requirements.txt. I don't really like this file, since it does not cover all possible dependencies. That's why we have a shell.nix. And it may also have irrelevant leftover dependencies from things you have installed but are no longer using. Furthermore its "pinning" is only skin-deep, it relies on PyPi to enforce versions. This is what having a shell.nix or default.nix is better. Ultimately you can do this to allow your downstream consumers of your package to decide how they want to run your code.

Also there's test dependencies and development dependencies. All of these can be encoded: https://stackoverflow.com/questions/28509965/setuptools-development-requirements

If you are using dependency_links, you need to also use --process-dependency-links flag when you use pip install . or pip install -e .. However this form of non-PyPi dependencies is deprecated, but I do not have information on what the alternative is.

When using Git submodules, you can get dependency_links to point to local URIs.

Pytest

If you are using pytest, you need to do a few more things.

Create setup.cfg with:

[aliases]
test=pytest

Inside the setup.py add this before the setup():

needs_pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
pytest_runner = ['pytest-runner'] if needs_pytest else []

Later inside setup():

setup_requires=[] + pytest_runner
tests_require=['pytest']

This ensures that pytestrunner is only needed during development. And pytest is needed during tests.

Inside your default.nix you only need this:

checkInputs = (with python.pkgs; [
  pytest
  pytestrunner
]);
checkPhase = ''
  pytest tests
'';