python_packaging_cheatsheet.md

Packaging cheat sheat

A generic set of commands for PyPi and conda packaging:

PyPI packaging

See: https://packaging.python.org/tutorials/packaging-projects/

Create conda environment with latest versions of setuptools, wheel and twine: conda create -n pypi-packaging setuptools wheel twine

Activate conda environment: conda activate pypi-packaging

To generate distribution package: python setup.py sdist bdist_wheel

Create a conda test environment: conda create -n test-sandbox <dependencies> conda activate test-sandbox

To test installing package: pip install dist\<wheel file.whl>

Reactivate conda environment: conda activate pypi-packaging

To upload all the archives under dist to test.pypi.org: twine upload --repository-url https://test.pypi.org/legacy/ dist/*

To install package from test.pypi.org: conda activate test-sandbox pip install --index-url https://test.pypi.org/simple/ <package name>

To upload on the real PyPI: twine upload dist/*

To install your package from the real PyPI: pip install <package name>

Including unit tests, add to setup.py, where point.to.a.tests.module contains TestCase class(es), e.g. <package name>.tests: test_suite="point.to.a.tests.module",

And run python setup.py sdist bdist_wheel test. This may run dependency unit tests as well. Package data, e.g. additional files for tests: package_data={'module.name': ['files*]},

Conda packaging

See: https://conda.io/docs/user-guide/tutorials/build-pkgs-skeleton.html

Create package in PyPi as above.

Create a conda environment for packaging:

• (windows) conda create -n conda-packaging m2-patch setuptools wheel pyyaml anaconda-client
• (linux) conda create -n conda-packaging setuptools wheel pyyaml anaconda-client

2023-03-08 Update:

# include boa to speed up
mamba create -n conda-packaging conda-build conda-verify setuptools wheel pyyaml anaconda-client boa -c conda-forge 
conda mambabuild <packagename>

Create directory for recipes: mkdir recipes cd recipes

Activate environment and create conda package skeleton: conda activate conda-packaging conda skeleton pypi <package name>

Download build1.sh and bld.bat.

Edit meta.yaml, example lines to change:

• Move hash to top of file: {% set sha256 = "xxx" %} Under source:: sha256: {{ sha256 }}
• Add licence under about: license_file: 'LICENSE'
• Remove unneeded lines in about: doc_url: and dev_url:
• Add github-id: extra: recipe-maintainers: - your-github-id-here

Build conda package: conda-build <package name>

Install into test environment, uninstalling previous pip installed instance first: conda activate test-sandbox pip uninstall <package name> conda install --use-local <package name>

Check installed: conda list

Perform any tests on package.

Build for different versions: conda-build --python 2.7 <package name> conda-build --python 3.4 <package name> conda-build --python 3.5 <package name>

Create directory dist in recipes, and convert for different architectures (does not work on win-64 due to conda bug): conda convert --platform all -o dist`

Create account on anaconda.org and login: anaconda login

Upload to anaconda: anaconda upload <output files>

To install from anaconda.org: conda create -n <env name> <package name> --channel https://conda.anaconda.org/<user name>

Update: 2020-03 build1.sh and bld.bat not needed for pure python

variants (e.g. different python versions) can be done using conda_build_config.yaml in recipes dir, e.g.:

python:
    - 3.6
    - 3.7

Updating versions

Bump version in source files and README.

Bump version in setup.py

Repackage for PyPi and upload:

conda activate pypi-packaging
python setup.py sdist bdist_wheel
twine upload dist/*

Get hash, go to https://pypi.org/project/<package name>/<version>/#files and click SHA256.

Bump version and update hash in recipes/<package name>/meta.yaml.

Build for conda (in recipes directory) using linux, convert and upload (do anaconda login if not yet logged in):

conda activate conda-packaging
for i in "" "--python 2.7" "--python 3.4" "--python 3.5"; do conda-build $i <package name>; done
conda convert --platform all <conda-bld package location with py* wildcard> -o dist
anaconda upload <conda-bld package location with py* wildcard>
anaconda upload dist/**/*<version>*

Update 2020-04-11: The above PyPi upload from conda build. To avoid uploading test versions to PyPi:

1. To build from local tarball produced by setuptools, get hash `sha256 dist/.tar.gz'
2. Change url in meta.yaml, suggest commenting out original url and adding this:

# to build with local tarball built by setuptools in dist directory, use following url
# use sha256sum to get hash
  url: ../../dist/{{ name }}-{{ version }}.{{ file_ext }}

Clean PyPi by deleting <package>.egg-info/, otherwise changes in packaged files may be cached.