ce-test-recipes-readme.md

About

This repo ce-test-recipes is a test repo for uploading to test pypi using concourse so we can prepare for the move from Jenkins to concourse for deployment when we are ready.

Using test pypi

https://packaging.python.org/guides/using-testpypi/

We all need to create accounts on test pypi so we can upload the package.

We need to remember to specify the --repository-url option or set the TWINE_REPOSITORY_URL environment variable when uploading so we don't affect pypi.

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

What Jenkins is doing

Have a look at the Jenkinsfile for what Jenkins does for us when we tag the package. Specifically the one step (formatted to make it easier to read):

sh "docker run --rm -e TWINE_USERNAME -e TWINE_PASSWORD -v ${WORKSPACE}:/src:rw --workdir /src python:3 /bin/bash -c \
    \"apt-get update && \
      apt-get install -y git && \
      pip install -q twine && \
      python setup.py bdist_wheel --universal && \
      twine upload dist/* && \
      rm -rf dist build *.egg-info versioneer.pyc\""

It uses the python 3 docker image, installs git and twine, creates a python binary distribution wheel file (--universal means the package works for both python 2 and 3) and use twine upload to upload to pypi.

Uploading to pypi manually

Now that you understand what Jenkins is doing, it is good to know how to do this manually in case concourse or Jenkins didn't work.

1. Do a fresh check out to have a clean environment:

   cd /tmp/
   git clone https://github.com/karenc/ce-test-recipes.git
   cd ce-test-recipes
   

2. Tag the repo and push to origin (just pick one that doesn't already exist):

   git tag -a v0.0.1
   git log --graph --oneline --decorate  # check origin/master and tag: v0.0.1 are on the same line
   git push origin v0.0.1
   

   The general rule I follow for the commit message is to have Release v0.0.1 and then the content of git log --format='- %s' --reverse v0.0.0.. in the body of the commit.

   Look at the release on github: https://github.com/karenc/ce-test-recipes/releases/tag/v0.0.1

3. Create a python 3 virtualenv and activate it:

   python3 -m venv py3
   . ./py3/bin/activate
   

   which python should show /tmp/ce-test-recipes/py3/bin/python

4. Install twine and wheel:

   pip install twine wheel
   

5. Build the binary distribution:

   python setup.py bdist_wheel --universal
   

   ls dist should show something like ce_test_recipes-0.0.1-py2.py3-none-any.whl

6. Upload the new version to test pypi:

   twine upload --repository-url https://test.pypi.org/legacy/ dist/*
   

   You will need to put in your username and password for test pypi.

7. Now check on test pypi for the version you just uploaded:

   https://test.pypi.org/project/ce-test-recipes/

Uploading to pypi using concourse

Our target is to use concourse to upload to pypi instead of using Jenkins. What concourse needs to do is similar to Jenkins but everything is configured using the pipeline.yml. There is nothing on the server except to read the logs and trigger jobs manually.

Have a look at concourse/dist-upload/pipeline.yml:

First we need to declare all the resources for the pipeline. The only resource is the git repo:

resources:
  - name: ce-test-recipes
    type: git
    source:
      uri: https://github.com/karenc/ce-test-recipes.git
      tag_filter: '*'

The tag_filter: '*' line is for getting only tags.

The first step of the job is to get the repository and specify how the job is triggered:

jobs:
  - name: dist upload
    public: true
    serial: true
    plan:
      - get: ce-test-recipes
        trigger: true

Concourse pipelines can be triggered by a resource. Details can be found here in the concourse tutorial.

This job should be automatically triggered every time there is a new tag pushed to github.

The second (and last) step of the job is to create the binary distribution and upload it to pypi:

      - task: twine upload
        config:
          platform: linux
          image_resource:
            type: docker-image
            source:
              repository: python
              tag: 3

          inputs:
            - name: ce-test-recipes

          run:
            path: /bin/bash
            args:
              - -c
              - |
                cd ce-test-recipes && \
                pip install -q twine && \
                python setup.py bdist_wheel --universal && \
                twine upload dist/*
          params:
            TWINE_USERNAME: ((twine-username))
            TWINE_PASSWORD: ((twine-password))
            # upload to test pypi
            TWINE_REPOSITORY_URL: https://test.pypi.org/legacy/

This step uses a docker image python:3 from https://hub.docker.com. It is going to use the output from the previous step ce-test-recipes as one of the inputs. The run: part just does something similar to Jenkins or what we would do manually. params: contains the environment variables we want to set for the docker container. ((twine-username)) and ((twine-password)) are variables that you can pass in. There are 3 ways to pass the variables in:

1. Pass the variables in as options to fly (not tested):

   -v twine-username=karen_chan -v twine-password=xxxxx
   

2. Create a yaml file with all the variables, e.g. vars.yml:

   twine-username: karen-chan
   twine-password: xxxxx
   

   and add this option to fly set-pipeline:

   -l vars.yml
   

3. Fetch the variables dynamically from Credentials Manager (not tested). There is nothing special to do here except the server needs to be configured. See Secret Parameters with Credentials Manager in the concourse tutorial.

To add the pipeline to concourse:

fly -t <concourse-instance> set-pipeline -p ce-test-recipes-pypi -c concourse/dist-upload/pipeline.yml -l concourse/dist-upload/vars.yml

Unpause the pipeline.

Push a tag to ce-test-recipes and wait for the job to start.