Fully-Managed Python Package using Poetry

github-cicd-workflow.yml

# Instructions:
#
# 1. Save this file as `{root}/.github/workflows/cicd.yml`.
# 2. Set the `PYPI_USER` and `PYPI_PASS` secrets in your GitHub CI config.
# 3. Update the `PYPACKAGE_NAME` declaration to the name you want to use in PyPi.
#
# Public Gist: https://gist.github.com/aaronsteers/f3eec4db94d3db1de78b76b3318e9e33

name: Python CI/CD
env:
  PYPACKAGE_NAME: YOUR-NAME-HERE 
on:
  push:
    branches:
      - "**"
    paths-ignore:
      - docs/**.md
jobs:
  python_build_and_test:
    runs-on: ubuntu-latest
    strategy:
      max-parallel: 4
      matrix:
        python-version:
          # - 3.7
          - 3.8
    steps:
      - name: Clone git repo
        uses: actions/checkout@v1
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v1
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install pipx and poetry
        run: |
          python3 -m pip install pipx
          python3 -m pipx ensurepath
          pipx install poetry
      - name: Install package using poetry
        run: |
          poetry install
      # - name: Lint with flake8
      #   run: |
      #     # stop the build if there are Python syntax errors or undefined names
      #     poetry run flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
      #     # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
      #     #poetry run flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

  pypi_publish:
    runs-on: ubuntu-latest
    strategy:
      max-parallel: 4
      matrix:
        python-version:
          - 3.7
    needs:
      - python_build_and_test
    steps:
      - name: Clone git repo
        uses: actions/checkout@v1
      - name: Set up Python (v${{ matrix.python-version }})
        uses: actions/setup-python@v1
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install pipx and poetry
        run: |
          python3 -m pip install pipx
          python3 -m pipx ensurepath
          pipx install poetry
      - name: Push to PyPi (build number ${{ github.run_number }})
        env:
          BRANCH_NAME: ${{ github.ref }}
          PYPI_USER: ${{ secrets.PYPI_USER }}
          PYPI_PASS: ${{ secrets.PYPI_SECRET }}
        run: |
          if [[ "$BRANCH_NAME" == *master ]]
          then
            poetry version --short
          else
            poetry version $(poetry version --short)-dev.$GITHUB_RUN_NUMBER
            poetry version --short
          fi
          echo -e "\nPublishing to version ref '$(poetry version --short)'...\n\n"
          poetry config http-basic.pypi $PYPI_USER $PYPI_PASS
          poetry publish --build
      - name: Wait up to 3 minutes for PyPi availability
        run: |
          pwd
          ls -la
          export VER=$(poetry version --short)
          export PIPERR=$(pip install $PYPACKAGE_NAME==$VER 2>&1)
          echo "Checking for PyPi availability of version $VER"
          if [[ $PIPERR == *"$VER"* ]]; then { echo "Yes"; } else { echo "Not yet found..."; sleep 30; } fi;
          export PIPERR=$(pip install $PYPACKAGE_NAME==$VER 2>&1)
          if [[ $PIPERR == *"$VER"* ]]; then { echo "Yes"; } else { echo "Not yet found..."; sleep 30; } fi;
          export PIPERR=$(pip install $PYPACKAGE_NAME==$VER 2>&1)
          if [[ $PIPERR == *"$VER"* ]]; then { echo "Yes"; } else { echo "Not yet found..."; sleep 30; } fi;
          export PIPERR=$(pip install $PYPACKAGE_NAME==$VER 2>&1)
          if [[ $PIPERR == *"$VER"* ]]; then { echo "Yes"; } else { echo "Not yet found..."; sleep 30; } fi;
          export PIPERR=$(pip install $PYPACKAGE_NAME==$VER 2>&1)
          if [[ $PIPERR == *"$VER"* ]]; then { echo "Yes"; } else { echo "Not yet found..."; sleep 30; } fi;
          export PIPERR=$(pip install $PYPACKAGE_NAME==$VER 2>&1)
          if [[ $PIPERR == *"$VER"* ]]; then { echo "Yes"; } else { echo "Not yet found..."; sleep 30; } fi;
          export PIPERR=$(pip install $PYPACKAGE_NAME==$VER 2>&1)
          if [[ $PIPERR == *"$VER"* ]]; then { echo "Yes"; } else { echo "Not found. Giving up. Last message from PyPi was $PIPERR"; exit 1; } fi;

poetry-migration-instructions.md

Instructions to Migrate Python Packages to Poetry and Full CI/CD Management

Install poetry

If you have not installed poetry already on your local machine:

python3 -m pip install pipx
python3 -m pipx ensurepath
pipx install poetry

Add PyPi Creds to CI Platform

In order to publish to PyPi, your CI environment will need access to creds - either a username/password combo or an API key.

Open your CI environment config and add the following as secrets:

• PYPI_USER - the identity to use when auth-ing with PyPi
• PYPI_SECRET - the secret key or password used for PyPi

Initialize Your Repo with Poetry

1. If you have not already done so, open a new branch on your repo (for example: feature/poetry).
2. From the root of the repo, run poetry init and answer the one-time prompts. If you have a setup.py file, it may be helpful to open that file before starting the process. (You will later be able to change these answers in pyproject.toml.)
   1. When asked for a version number, do not accept the default (0.1.0), but instead consult your VERSION file (if applicable) and provide a version that is (at least) a minor version up from that starting version.
   2. When asked for python compatibility, be sure to use ^3.6 (or earlier) if you want to keep support for early python versions. (You might find prior version support definitions in setup.py.)
   3. When asked for the dependencies, consult your requirements.txt file (if applicable). Also consult your setup.py file for any additional requirements not listed in requirements.txt.
   4. When asked for development dependencies, consult your setup.py file for any reference to tests_require.
3. Check your setup.py file for any references to entry_points or console_scripts, which are the executable CLI commands of the package.
   1. If applicable, add the followiing section to your pyproject.toml file:

      [tool.poetry.scripts]
      runme = "mylibrary.cli_file:main_func"

4. Check your setup.py file for any references to extras_require - which represent optional features which require additional upstream dependencies. If applicable:
   1. If you have not already done so, add the optional packages in the same way as you normally would, either by declaring them during poetry init or with poetry add package1 package2 etc.
   2. After all optional packages have been added, update the following section to your pyproject.toml file, using the "optional = true" syntax and referencing the optional packages in the extras section:

      [tool.poetry.dependencies]
      # These packages are mandatory and form the core of this package’s distribution.
      mandatory = "^1.0"
      
      # A list of all of the optional dependencies, some of which are included in the
      # below `extras`. They can be opted into by apps.
      psycopg2 = { version = "^2.7", optional = true }
      mysqlclient = { version = "^1.3", optional = true }
      
      [tool.poetry.extras]
      mysql = ["mysqlclient"]
      pgsql = ["psycopg2"]

   3. Important: extras must be defined in in pyproject.toml using all lowercase or they will not work properly. However, during installation these are equivalent: pip install mypackage[mysql] is the same as pip install mypackage[MySQL].

Delete Files No Longer Needed

The following files are replaced by pyproject.toml and should be deleted from your repo:

1. MANIFEST.in
2. requirements.txt
3. setup.cfg
4. setup.py
5. VERSION