Skip to content

Instantly share code, notes, and snippets.

@maximlt

maximlt/project_config.md

Last active March 18, 2024 19:49
Show Gist options
  • Save maximlt/629f89e2c6dddd688a23de9ea869de89 to your computer and use it in GitHub Desktop.
Save maximlt/629f89e2c6dddd688a23de9ea869de89 to your computer and use it in GitHub Desktop.
Download ZIP
Creating a project base
Raw
project_config.md

Steps to create a good project base for developing in a conda environment

Step-by-step (uncomplete) tutorial for setting up a base Python library given the following requirements:

  • Use conda (instead of pipenv or others) because this is both a package manager and an environment manager, and installing the Python scientific stack (Numpy, Pandas, Scipy, Matplotlib, etc.) is straightforward
  • Use Visual Studio Code (instead of PyCharm, Spyder or others) because it's free, runs on Windows and is one of the mostly used IDE
  • Document and automate as many production steps as possible including linting (flake8), formatting (black), packaging (setup.py, setup.cfg), versionning (git), testing (pytest, pytest-cov, tox), documenting (sphinx, readthedocs), building (setuptools) and distributing (twine, keyring)
  • Include IPython Notebooks and have them tested (pytest-nbval)

Table of content

Create a new conda environment

If possible, use conda-forge only because there are more packages available compared to the defaults channel and the less pip install the better.

  1. Execute the following (this is equivalent to conda create -n test --channel conda-forge --strict-channel-priority)
conda create -n project
conda activate project
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict
  1. Execute the following in the active environment to check the setup conda config --show channels channel_priority
channels:
  - conda-forge
  - defaults
channel_priority: strict

References:

Create a working repository

mkdir project
cd project

Set up Visual Studio Code

Start to set things up

  1. Open VSCode
  2. Open the project folder
  3. Find Select Interpreter in the command palette (CTRL + SHIFT + P) and pick the right conda environment

Notes:

  • Do NOT save as a .code-workspace file in the repo, this is useless for the project itself. Custom settings are automatically saved in .vscode/settings.json

Automatic Numpy doctrings

  1. Install the extension AutoDocstring
  2. Go to the workspace settings and set Auto Docstring: Docstring Format to numpy
  3. Go to settings.json and add "files.trimTrailingWhitespace": true because whitespaces are added between each section

Configure pytest

VSCode has the following issues with integrating pytest:

  • VSCode display the test output in a dedicated prompt Python Test Log where the colors are not rendered :(
  • Debugging tests doesn't work when pytest-cov is installed, it needs to be deactivated with --no-cov (TODO: Add links and more info)
  • Automatic tests discovery doesn't work so well with conda (TODO: Add links to issues)

Besides that, pytest seems to be a really powerful and flexible command line tool. It may be worth learning how to use it directly before using it through VSCode.

  1. Create a tests folder at the root of project (it could also be within the package folder)
  2. Add a testing script (e.g. test_mod1.py or mod1_test.py) including some tests (e.g. test_foo(), test_bar()).
  3. Add an \_\_init__.py file next to the tests file (tip from the official VSCode doc here.
  4. Find Python: Configure Tests in the command palette
  5. Select the tests folder
  6. Select pytest and install it with conda (again, the less pip install the better)
  7. If required, run Python: Discover Tests to ask pytest to discover all our test_.py/_test.py test files

Note: Tests can also be run directly from the command line with pytest. python -m pytest does almost the same thing, except that it adds the current directory to sys.path which might be useful to find the tested package. Otherwise the good practice (outside of VSCode) seems to be (see here) to pip install -e . the package to test so that it's available to pytest.

Add linting and formatting

Some info here: http://books.agiliq.com/projects/essential-python-tools/en/latest/linters.html

  1. Open commande palette
  2. Find Python: Select linter
  3. Install flake8
  4. Go to settings.json and add (see https://github.com/psf/black/blob/master/.flake8) to adapt flake8 for black
    "python.linting.flake8Args": [
        "--ignore=E203,E266,E501,W503",
        "--max-line-length=88",
        "--max-complexity=18",
        "--select=B,C,E,F,W,T4,B9"
    ],
  1. Find Format Document in the command palette
  2. Install black
  3. Go to settings.json and add "editor.formatOnSave": true, "files.trimFinalNewlines": true

Clean the file explorer

Go to settings.json and add (TODO: Add the complete content):

    "files.exclude": {
        "**/__pycache__": true,
        "**/*.pyc": {
            "when": "$(basename).py"
        },
        "**/.vscode": true,
        "**/.pytest_cache": true,
    }

References:

Git the repo

  1. Create a github repository (this is done directly on the website)
  2. Find Git: Initialize Repository
  3. Find Git: Add Remote, type in origin and the repo URL https://github.com/maximlt/project.git. This can also be done with the command line: git remote add origin https://github.com/maximlt/project.git
  4. Add a .gitignore file at the repo root TODO Add more stuff in the .gitignore
# Byte-compiled / optimized / DLL files
__pycache__/

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# Unit test / coverage reports
.tox/
.pytest_cache/

# Sphinx documentation
docs/_build/

# Jupyter Notebook
.ipynb_checkpoints

# Misc
.garbage/
.idea/

Package it

Create a setup.py file

  1. Create a setup.py file with the following code:
import pathlib
from setuptools import setup, find_packages

# The directory containing this file
HERE = pathlib.Path(__file__).parent

# The text of the README file
README = (HERE / "README.md").read_text()

# This call to setup() does all the work
setup(
    name="project",
    version="0.0.1",
    description="Project description",
    long_description=README,
    long_description_content_type="text/markdown",
    url="https://github.com/maximlt/project",
    author="Maxime Liquet",
    author_email="maximeliquet@free.fr",
    license="MIT",
    classifiers=[
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.7",
    ],
    packages=['project'],
    include_package_data=True,
    install_requires=[],
    entry_points={
        "console_scripts": [
            "project_cli=project.cli:main",
        ]
    },
)
  1. Add a README.md that will be nicely rendered on GitHub and TestPyPi and PyPi

Make versionning simpler

  1. Add __version__ = "0.0.12" to project/_init_.py
  2. Add this to setup.py
import pathlib
import codecs
import re

# The directory containing this file
HERE = pathlib.Path(__file__).parent

def read(*parts):
    # with codecs.open(os.path.join(HERE, *parts), "r") as fp:
    with codecs.open(HERE.joinpath(*parts), "r") as fp:
        return fp.read()

# Adapted from (1) https://packaging.python.org/guides/single-sourcing-package-version/
def find_version(*file_paths):
    version_file = read(*file_paths)
    version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]", version_file, re.M)
    if version_match:
        return version_match.group(1)
    raise RuntimeError("Unable to find version string.")

setup(
    ...,
    version=find_version("project", "__init__.py"),
    ...,
)

Notes:

  • There are many (many) ways to add the version number programmaticaly, this is just one (almost) simple way
  • bump2version seems to be an interesting automation tool

References:

Add a setup cfg file

[metadata]
# This includes the license file(s) in the wheel.
# https://wheel.readthedocs.io/en/stable/user_guide.html#including-license-files-in-the-generated-wheel-file
license_files = LICENSE

[bdist_wheel]
# This flag says to generate wheels that support both Python 2 and Python
# 3. If your code will not run unchanged on both Python 2 and 3, you will
# need to generate separate wheels for each Python version that you
# support. Removing this line (or setting universal to 0) will prevent
# bdist_wheel from trying to make a universal wheel. For more see:
# https://packaging.python.org/guides/distributing-packages-using-setuptools/#wheels
universal = 0

Notes:

Install it in develop mode in the current conda environment

There seems to be at least two ways to do that.

1

  • In the same repo run pip install -e . to install the package in develop mode.
  • Run pip uninstall project to uninstall it (if it doesn't work, try python setup.py develop --uninstall)

2

  • Go to the repo root and run conda-develop ., which will add a link to the repo in conda.pth (site-packages of the current conda env), and subsequently add this link to sys.path
  • To (not very succesfully) uninstall the package, run conda-develop -u .

Notes:

  • 2 doesn't seem to work so well. Check in the files conda.pth and easy-install.pth that the paths to the project repo are deleted with things go wrong. Otherwise it's added to sys.path automatically.
  • The advantage of 1 over 2 is that the packge appears in conda list. There may be some other differences. Installing conda-build with 2 doesn't solve the issue.

Add some docs with Sphinx

Some info here https://tutos.readthedocs.io/en/latest/source/git_rtd.html

  1. Run conda install sphinx
  2. Create a docs directory
  3. Run sphinx-quickstart from that directory
  4. Run conda install sphinx_rtd_theme
  5. Modify conf.py by adding 'sphinx_rtd_theme' to the extensions list and by defining html_theme = 'sphinx_rtd_theme'
  6. Add 'sphinx.ext.autodoc'to the extensions
  7. Add 'sphinx.ext.napoleon'to the extensions and napoleon_numpy_docstring = True to set Numpy DocStrings
  8. Add master_doc = 'index' for the index.rst file to be the main entry file
  9. Add this to import the project
import os
import sys
sys.path.insert(0, os.path.abspath('../../'))

import project
  1. Exemple of an index doc
Welcome to project's documentation!
===================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   installation
   api

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
  1. Example of an api doc (the docstring will be read automatically with autofunction)
.. _api:

API Documentation
=================

Foo
----

.. autofunction:: project.script.foo
  1. Run cd docs and make html to render the doc and adjust
  2. Commit and push
  3. Go to https://readthedocs.org/ and import the GitHub project
  4. Create a RTD project (same name hopefully) and <3 generate the docs <3

Notes:

  • Run sphinx-build -b html docs\source docs\build from the repo root to build the docs

References:

Test it

Create some tests

  • Add a tests repo

Test a Click app: https://stackoverflow.com/questions/52706049/how-to-test-a-python-cli-program-with-click-coverage-py-and-tox

Add coverage

From the VS Code doc:

Note If you have the pytest-cov coverage module installed, VS Code doesn't stop at breakpoints while debugging because pytest-cov > is using the same technique to access the source code being run. To prevent this behavior, include --no-cov in pytestArgs when > > debugging tests. (For more information, see Debuggers and PyCharm in the pytest-cov documentation.)

Still, let's give it a try. If it doesn't work, check this out: microsoft/vscode-python#693 TODO: check it

  1. Run conda install pytest-cov
  2. Run python -m pytest -v --cov-branch --cov=project --cov-report=html tests/ (not so sure about the effect of --cov-branch)
  3. Add this to .gitignore
coverage_html_report/
htmlcov/
.coverage
  1. Add this to settings.json
        "**/coverage_html_report": true,
        ".coverage": true

References:

Add tox

TODO: Check updates about tox-conda because it is likely the issues described below will be fixed in the future.

  1. Run conda install tox
  2. Add a tox.ini file
[tox]
envlist = py36, py37

[testenv]
deps =
    pytest
    pytest-cov

changedir = {toxinidir}/tests
commands = python -m pytest --cov={envsitepackagesdir}/project

"**/.tox": true, 3. tox will work well with the current Python environment (e.g. py37), but it won't work with another one (e.g. py36). To fix, execute conda create -p C:\Python36 python=3.6 to create a conda environment that will be detected by tox. To delete that environment, execute conda remove -p C:\Python36 --all (it cannot be found by name)

Notes:

[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"

[tool.flit.metadata]
module = "project"
author = "Maxime Liquet"
author-email = "maximeliquet@free.fr"
description-file = "README.md"
home-page = "https://github.com/maximlt/project"
classifiers = ["License :: OSI Approved :: MIT License"]
dist-name="projectxyxyxy"

[tool.flit.metadata.requires-extra]
test = ["pytest", "pytest-cov", "tox"]
doc = ["sphinx", "sphinx_rtd_theme"]

[tool.flit.metadata.urls]
Documentation = "https://xyxyxy.readthedocs.io/en/latest/"

and tox.ini should look like

[tox]
envlist = py37

# To create a source distribution there are multiple tools out there and with PEP-517 and PEP-518
# you can easily use your favorite one with tox. Historically tox only supported setuptools, and
#always used the tox host environment to build a source distribution from the source tree. This is
# still the default behavior. To opt out of this behaviour you need to set isolated builds to true.
isolated_build = True

[testenv]
deps =
    pytest
    pytest-cov

changedir = {toxinidir}/tests
commands = python -m pytest --cov={envsitepackagesdir}/project

Add notebooks and test them

  1. Create a notebooks directory
  2. Run conda install jupyterlab (this will install a LOT of packages)
  3. Make sure project is installed (pip install -e .)
  4. cd to the directory and launch a notebook with jupyter lab
  5. Write some code using the package project and save the notebook (e.g. example.ipynb)
  6. Run conda install nbval (a newer version might be available on PyPi, but this is a pytest plugin so let's stick to conda as pytest was installed from there)
  7. Run pytest --nbval -v example.ipynb (-v for verbose mode) or pytest --nbval -v notebooks/

Notes:

  • --nbdime is possible and should provide nice diffs, but it requires an nbdime install (not tested, probably not required at all)

References:

Push with a tag and release on GitHub

  1. Find Git: Create Tag in the command palette
  2. Add a tag like 0.0.1 and a note like 0.0.1 Release
  3. Find Git: Push (Follow Tags) in the command palette and push
  4. Go to the GitHub repo, releases tab and Draft a new release
  5. Add the same tag 0.0.1, some notes and save

Build it

  1. If some more files need to be included in the distribution (e.g. tests, docs, etc.) add a MANIFEST.in file for distutils to know which files to include/exclude (see https://stackoverflow.com/questions/41661616/what-is-the-graft-command-in-pythons-manifest-in-file) and add include_package_data=True, to setup.py
# MANIFEST.in
recursive-include tests *.py

Notes:

  • If setup.py uses a README.md file for defining the long description of a package, that file must be included in MANIFEST.in. If not that would for instance break tox.
  1. Run python setup.py sdist bdist_wheel (add clean --all to delete the build output) to create a source distribution and a wheel to dist/

Notes:

  • TODO: Whatever is added to MANIFEST.in doesn't seem to be included in site-packages, why?

References:

Publish it

TODO: Mention that it's possible to define environment variables (e.g. TWINE_USERNAME and TWINE_PASSWORD) that are super convenient.

  1. Install twine with conda install twine
  2. Check if building went fine with twine check dist/*
  3. Install keyring to handle the PyPi password with conda install keyring
  4. Set the passwords with keyring:
    • For TestPyPi: keyring set https://test.pypi.org/legacy/ my-testpypi-username and type in my-testpypi-password
    • For PyPi: keyring set https://upload.pypi.org/legacy/ my-pypi-username and type in my-pypi-password
  5. Publish
    • To TestPyPi: run twine upload --skip-existing --repository-url https://test.pypi.org/legacy/ dist/*
    • To PyPi: run twine --skip-existing upload dist/*

Notes:

  • --skip-existing: continue uploading files if one already exists (this seems to be useful when there are several package distributions in /dist)
  • A .pypirc file could be saved at C:\Users\myuserspace (at least on Windows 10), but that's not particulary useful
[distutils]
index-servers =
   pypi
   testpypi

[pypi]
username: mypypiusername

[testpypi]
repository: https://test.pypi.org/legacy/
username: mytestpypiusername

References:

Install it

  • From TestPyPi: pip install -U --index-url https://test.pypi.org/simple/ projectxyxyxy
  • From PyPi: pip install -U project

Notes :

  • U, --upgrade: upgrade all packages to the newest available version, without that switch it'll say the package is already installed and exit.

Export an environment.yml file

  • Run conda env export > environment.yml
  • It can be saved on GitHub to allow developpers to easily set up a development environment

Not used but interesting

Add git hooks with pre-commit

  1. Run conda install pre_commit
  2. Create a file .pre-commit-config.yaml at the repo root
  3. Add this to the file
repos:
-   repo: https://github.com/psf/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.7

Publish it with flit

Status: not working

Unfortunately, it seems a little complicated to use flit in a conda environment. flit install --pth-file (develop mode) breaks conda list. And it installs all the dependencies with pip while conda would be prefered. There may be some improvement in the future, so check that because this is a really nice tool to install/build/publish a library.

  1. Run conda install flit
  2. Run flit ini and fill in the answers
  3. pyproject.tml is created, in the end it should look like this:
[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"

[tool.flit.metadata]
module = "project"
author = "Maxime Liquet"
author-email = "maximeliquet@free.fr"
description-file = "README.md"
home-page = "https://github.com/maximlt/project"
classifiers = ["License :: OSI Approved :: MIT License"]

[tool.flit.metadata.requires-extra]
test = ["pytest", "pytest-cov", "tox"]

[tool.flit.metadata.urls]
Documentation = "https://flit.readthedocs.io/en/latest/"
  1. Add a docstring at the beginning of init.py and `version == "0.0.1"
  2. Add a .pypirc file at C:\Users\maxim (that's the trick, not in the project repo)
[distutils]
index-servers =
   pypi
   testpypi

[pypi]
username: maximelt

[testpypi]
repository: https://test.pypi.org/legacy/
username: maximelt
  1. Run conda install keyring to handle the PyPi password
  2. Run flit --repository testpypi publish to publish the package on test.pypi (the password must be entered only once, hopefully)
  3. TODO: See how to publish on PyPi

References:

Base project with cookiecutter

Let's try this cookiecutter template: https://github.com/ionelmc/cookiecutter-pylibrary

It requires cookiecutter and tox. I add tox-conda to (try to) set up a developing environment in conda. conda install cookiecutter tox tox-conda

General references

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment