coordt/release-tooling.md

## release-tooling.md

      
    Raw
  

              release-tooling.md
            
          
    Releasing Software

What is a software release?

A release is the immutable packaging of a software product. Software releases are an essential component of modern software development and the software development life cycle for two main reasons:

They are intentional. Someone decided what changes are required to give value and when the changes are ready for others to use.
They include context. Information about the difference between the code at two points is conveyed. Whether it is simply by the version indicating the scope or including additional artifacts such as a changelog, the user has more information than "something changed."

The developer plays a key role in a software release because a thoughtful decision is required.
What is release management?

Release management is how releasing software fits into your development process. Some things to consider:

Release Policies: The definition of release types, standards, and governance requirements.
Release Unit: The set of artifacts released together.
Release Pipeline: A repeatable workflow process that includes human and automated activities and follows the release policies.
Release Value Stream: Extra processes that add or create value across the release pipeline.
Pre-releasing: Ability to release software before finalizing the code for integration testing with other services.
Parallel development: The ability for parallel and non-blocking development on the same repository.

An example release management processs

This section provides a concrete example of a working software relase management system.
Release policies

A software release consists of release unit(s) tagged with a version number.
Version numbering

A software release's version number should:

convey some contextual information.
easily link the resulting artifacts and the git commit used to generate the artifacts.

Versioning methods

Software that delivers features ad-hoc, will use a semantic versioning system. The version will provide some contextual information about each release's difference from another version.
Software that delivers regular, periodic updates, will use a calendar versioning system. The version will provide information about how current an installed version is from the current version.
Software that does not satisfy the above conditions should use the first 7 characters of the git commit SHA for version reporting. A form of calendar version could also be used if the date-time contextual information is useful.
Linking the version to the git commit

For semantic and calendar versioning, tag the git commit that contains all the finished code and documentation and should be used to generate the final artifacts with the version number. Continuous integration processes can use the tag as a trigger to automate the packaging.
Semantic versioning

We will use the format MAJOR.MINOR.PATCH for final releases and MAJOR.MINOR.PATCH+BRANCH_NAME-DEV for development releases while on feature branches.
For final releases, increment the:

PATCH version when you make backwards compatible bug fixes.
MINOR version when you add functionality in a backwards compatible manner
MAJOR version when you make incompatible API changes

For development releases, only the DEV version is incremented. The MAJOR, MINOR, and PATCH segments indicate the version this development release is based on.
Release unit


Package for libraries
Docker container for services

Release pipeline

A git tag on a commit in the main/master branch using the version number as the label indicates a release.
The tagged commit ideally should contain all artifacts (changelog, release notes, test reports, package metadata) pertaining to the release, but they may be included in previous commits.
All subsequent commits on that branch are considered part of the next release.
The recommended method is:

Development is done in one or more feature branches
A pull request for the feature branch is created and reviewed
When the code in the pull request is finished and could be released, it is merged into main/master
The merged branches are deleted
A developer checks out the latest main/master branch
The developer runs a release command (see below) to trigger the appropriate release type.

We will set up automated tooling below.
Release value stream

There are two parts of the value stream:

Building the artifacts required before the tag.
Building the artifacts as a result of the tag.

The first part requires input from the developer: the "type" of release (major, minor, patch). The method of getting this information should be:

Easy to set up
Easy to maintain
Easy to learn
The easiest way to release
Resilient for errors and mistakes

We feel a Makefile with commands for releasing major, minor or patch versions fills this role the best, based on the above criteria.
The second part is run by a continuous integration platform, such as GitHub Actions or Azure Pipelines. These pipelines can detect new tags in the repository and automate additional steps.
Pre-releasing

The ability to create a non-final software release for beta testing or integration testing is useful to prevent regressions or confirm a bug was fixed in a real-world scenario.
When on a feature branch, developers can release a development version. The resulting format is <BASE VERSION>+BRANCH_NAME-DEV. For example a development release on a branch named "bug-fix-451" that is branched from version 1.2.3 would release versions 1.2.3+BUGFIX451-0 and 1.2.3+BUGFIX451-1 and so on.
Including the branch name and the version the branch is based off allows parallel development without conflicts.
Parallel development

Working on a feature should not block fixing a bug in the latest release. Development releases do not conflict due to naming conventions.
Conflicts for final release is managed using git itself via the automated tools. Since the release pipeline allows the developer to specify the release type and not the version, the tools pull down the latest commits and increments the version according to the latest data in git.
If two developers tried to do a final release at the exact same time, the first one who pushes to origin would cause a git error for the other when they pushed.
Version management implementation

Store the version in your package

We are going to store the version in our code for convenience. We'll add a __version__ attribute to our package for this.
Modify the __init__.py file in your package such as for mypackage:
mypackage/_init_.py:
"""This is my package; there are many like it but this one is mine."""

__version__: str = "0.1.0"
This allows us to:
>>> import mypackage
>>> print(mypackage.__version__)
0.1.0
Update our packaging information

Our packaging will reference the __version__ when it builds in the [metadata] section to the setup.cfg file in our repo:
setup.cfg:
[metadata]
name = mypackage
version = attr: mypackage.__version__
description = "This is my package; there are many like it but this one is mine."
long_description = file:README.md
long_description_content_type = text/markdown
The attr: mypackage.__version__ on line 3 tells setuptools to use the __version__ attribute that we set up previously, making it easier to manage our release version.
Release version management via tools

We will do the following to manage our versions:

Use bump2version for version management.
Add configuration to setup.cfg to configure how to manage our version progressions.
Add a Makefile to automate the process

Bumpversion (bump2version)

Bump version is a small command line tool to simplify releasing software by updating all version strings in your source code by the correct increment. It also creates commits and tags.
First you need to add bump2version to your development requirements.

Note
If you don't manage development requirements separately from production requirements, then you can put it in the production requirements; it just isn't required for your code to work.

setup.cfg:
[bumpversion]
current_version = 0.1.0
commit = True
commit_args = --no-verify
tag = True
tag_name = {new_version}
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\+[-_\w]+?-(?P<dev>\d+))?
serialize =
    {major}.{minor}.{patch}+{$BRANCH_NAME}-{dev}
    {major}.{minor}.{patch}
message = Version updated from {current_version} to {new_version}

[bumpversion:file:mypackage/__init__.py]
Here is what the lines mean:


current_version: The current version of the software package before bumping. Documentation.


commit: Create a commit after bumping. Documentation.


commit_args: Extra arguments to pass to commit command. --no-verify prevents git commit hooks from running during commit. Documentation.


tag: Create a tag named as the tag_name setting. Documentation.


tag_name: Set the name of the tag to the new version. Documentation.


parse: Regular expression (using Python regular expression syntax) on how to find and parse the version string.


serialize: Template specifying how to serialize the version parts back to a version string. bumpversion will try the serialization formats beginning with the first and choose the last one where all non-zero values can be represented.
In this case, it will render the version as {major}.{minor}.{patch} if the dev value is 0. {$BRANCH_NAME} references an environment variable that is covered later. Documentation.


message: The commit message to use when creating a commit. Documentation.


[bumpversion:file:mypackage/__init__.py]: This tells bumpversion to look in the mypackage/__init__.py for the current version string update the string to the new version.   Documentation.


Automation with make

This creates a set of commands:

make release-dev Release a new development version only if you are on a feature branch. (1.1.1 to 1.1.1+branchname-1)
make release-patch Release a new patch version. (1.1.1 to 1.1.2)
make release-minor Release a new minor version. (1.1.1 to 1.2.0)
make release-major Release a new major version. (1.1.1 to 2.0.0)
make release-version <new version number> Set version to <new version number> and release.

Makefile:
.DEFAULT_GOAL := help
BRANCH_NAME := $(shell git rev-parse --abbrev-ref HEAD)
SHORT_BRANCH_NAME := $(shell echo $(BRANCH_NAME) | cut -c 1-20)
PRIMARY_BRANCH_NAME := master
BUMPVERSION_OPTS :=

help:
    @grep '^[a-zA-Z]' $(MAKEFILE_LIST) | sort | awk -F ':.*?## ' 'NF==2 {printf "\033[36m  %-25s\033[0m %s\n", $$1, $$2}'

release-dev: RELEASE_KIND := dev
release-dev: do-release  ## Release a new development version: 1.1.1 -> 1.1.1+branchname-1

release-patch: RELEASE_KIND := patch
release-patch: do-release  ## Release a new patch version: 1.1.1 -> 1.1.2

release-minor: RELEASE_KIND := minor
release-minor: do-release  ## Release a new minor version: 1.1.1 -> 1.2.0

release-major: RELEASE_KIND := major
release-major: do-release  ## Release a new major version: 1.1.1 -> 2.0.0

release-version: get-version do-release  ## Release a specific version: release-version 1.2.3

#
# Helper targets. Not meant to use directly
#

do-release:
	@if [[ "$(BRANCH_NAME)" == "$(PRIMARY_BRANCH_NAME)" ]]; then \
		if [[ "$(RELEASE_KIND)" == "dev" ]]; then \
			echo "Error! Can't bump $(RELEASE_KIND) while on the $(PRIMARY_BRANCH_NAME) branch."; \
			exit; \
		fi; \
	elif [[ "$(RELEASE_KIND)" != "dev" ]]; then \
		echo "Error! Must be on the $(PRIMARY_BRANCH_NAME) branch to bump $(RELEASE_KIND)."; \
		exit; \
	fi; \
    export BRANCH_NAME=$(SHORT_BRANCH_NAME);bumpversion $(BUMPVERSION_OPTS) $(RELEASE_KIND) --allow-dirty; \
    git push origin $(BRANCH_NAME); \
    git push --tags;

get-version:  # Sets the value after release-version to the VERSION
	$(eval VERSION := $(filter-out release-version,$(MAKECMDGOALS)))
	$(eval BUMPVERSION_OPTS := --new-version=$(VERSION))

%: # NO-OP for unrecognized rules
	@:
The variables at the top are used to configure the behavior:

.DEFAULT_GOAL The target to run when no target is passed. If someone simply runs make it will show the help text.
BRANCH_NAME The current branch name.
SHORT_BRANCH_NAME The first 20 characters of the branch name. Keeps versions from getting too long.
PRIMARY_BRANCH_NAME The name of the branch for doing non-development releases.
BUMPVERSION_OPTS Additional options passed to bumpversion.

The Makefile's phony targets require more explanation. Some of the targets are not meant to be called directly.

help (lines 7-8) parses and displays the text after ## comments.
release-* (lines 10-22) creates the release type specified in the name.
do-release (lines 28-40) does the actual release work.
Lines 29-37 check the current branch to determine if you can do a development release or not.
Lines 38-40 run the commands to change the version, commit, tag and push the result.
Don't call this target directly!
get-version (lines 42-44) parses the version from the make command and updates the BUMPVERESION_OPTS variable to the correct option for bump2version.
% (lines 46-47) does nothing if an unrecognized target is passed to the make command. This is required so that the version passed in with release-version doesn't generate an error.

Updating a changelog when releasing

Now that we have a set of commands to automate a version change and software release, let's add additional value by updating a changelog using the git commits as the basis and the version-tagged commits as the basis for grouping commits into releases.
generate-changelog configuration

Add generate-changelog to your development requirements. Then install it.
pip install generate-changelog
You can generate the entire default configuration file by running:
generate-changelog --generate-config
But you can also just create a configuration that overrides the default values. We are going to do that here to highlight the changes we will make.
.changelog-config.yaml:
variables:
  changelog_filename: CHANGELOG.md
  repo_url: https://github.com/<username>/mypackage
The default configuration sets the changelog_filename variable. Since we are overriding the variables section, we need to include it.
We are adding the repo_url variable so we can add additional links in our templates.
Overriding the changelog templates

We are going to add links to each commit and a link to a diff between releases. Make sure the default location for extra templates .github/changelog_templates exists:
mkdir -p .github/changelog_templates
Now add two files, commit.md.jinja and version_heading.md.jinja:
commit.md.jinja:
- {{ commit.summary }} [{{ commit.short_sha }}]({{ repo_url }}/commit/{{ commit.sha }})
  {{ commit.body|indent(2, first=True) }}
  {% for key, val in commit.metadata["trailers"].items() %}
  {% if key not in VALID_AUTHOR_TOKENS %}
  **{{ key }}:** {{ val|join(", ") }}

  {% endif %}
{% endfor %}
The change in this template is on the first line. After the commit summary, the short git SHA is made into a link, using the repo_url variable as the root.
version_heading.md.jinja:
## {{ version.label }} ({{ version.date_time.strftime("%Y-%m-%d") }})

[Compare the full difference.]({{ repo_url }}/compare/{{ version.previous_tag }}...{{ version.tag }})
Line 3 is the addition to this template. It uses the repo_url variable to make a link to the difference between this version and the previous.
Update bumpversion configuration

When the changelog is generated, the latest commits are grouped under the title "Unreleased" and the diff link references the difference between the previous version and HEAD. These two things need replacing with the new version.
Bumpversion can do this for us. Add the following lines to setup.cfg:
setup.cfg addition:
[bumpversion:file(version heading):CHANGELOG.md]
search = Unreleased

[bumpversion:file(diff link):CHANGELOG.md]
search = {current_version}...HEAD
replace = {current_version}...{new_version}
Add changelog generation to release automation

We just need to make a few changes to our Makefile to enable changelog generation and optional editing of the changelog before commit and release.
Makefile:
Add (after line 5)
EDIT_CHANGELOG := $(shell if [[ -n $$EDITOR ]] ; then echo "$$EDITOR CHANGELOG.md" ; else echo "true" ; fi)
This  adds the command to edit the changelog if the EDITOR environment variable is set when $(call EDIT_CHANGELOG) is run within the do-release target. If EDITOR is not set, it puts in the true command, which causes it to move to the next step.
In the do-release target, change the lines:
    export BRANCH_NAME=$(SHORT_BRANCH_NAME);bumpversion $(BUMPVERSION_OPTS) $(RELEASE_KIND) --allow-dirty; \
    git push origin $(BRANCH_NAME); \
    git push --tags;
to
	git fetch -p --all; \
	generate-changelog; \
	$(call EDIT_CHANGELOG); \
	export BRANCH_NAME=$(SHORT_BRANCH_NAME);bumpversion $(BUMPVERSION_OPTS) $(RELEASE_KIND) --allow-dirty; \
	git push origin $(BRANCH_NAME); \
	git push --tags;
The three additional lines makes sure that your branch is up to date, generates the changelog and then brings up the changelog for editing (if your EDITOR environment variable is set).
Continuous integration and releasing

Nested PRs for parallel feature development