Tutorial and tips for GitHub Actions workflows
GitHub Actions is a CI/CD service that runs on GitHub repos.
Compared with Travis CI, GitHub Actions is:
- Easier
- More flexible
- More powerful
- More secure
- Workflows are YAML files stored in the .github/workflows directory of a repository.
- An Action is a package you can import and use in your workflow. GitHub provides an Actions Marketplace to find actions to use in workflows.
- A job is a virtual machine that runs a series of steps. Jobs are parallelized by default, but steps are sequential by default.
- To get started:
- Navigate to one of your repos
- Click the "Actions" tab.
- Select "New workflow"
- Choose one of the starter workflows. These templates come from actions/starter-workflows.
- Workflows can be triggered by many different events from the GitHub API. The
workflow_dispatchtrigger allows workflows to be triggered manually, with optional input values that can be referenced in the workflow. - GitHub provides an expression syntax for programmatic control of workflows. For example:
echo "SPAM_STRING=${{ format( 'Spam is short for {0} and is made from {1} by {2}', 'spiced ham', 'pork shoulder', 'Hormel' ) }}" >>"$GITHUB_OUTPUT"- Command:
echo "ENV_NAME=value" >>"$GITHUB_OUTPUT", likeecho "COLOR=green" >>"$GITHUB_OUTPUT" - Expression:
${{ }} - Function:
contains('this is a demo', 'demo')evaluates to Booleantrueformat('Spam is short for {0} and is made from {1} by {2}', 'spiced ham', 'pork shoulder', 'Hormel')
- Command:
A workflow file might look like this:
name: demo
on:
push:
branches: [demo]
pull_request:
workflow_dispatch:
env:
APP_NAME: "GitHub Actions demo workflow"
jobs:
simple:
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@main
- name: Verify the workspace context
run: echo 'Workspace directory is ${{ github.workspace }}'
- name: Run a simple echo command with a pre-set environment variable
run: echo 'Hello World, from ${{ env.APP_NAME }}'
- name: Set an environment variable using a multi-line string
run: |
echo "MULTI_LINE_STRING<<EOF" >>"$GITHUB_ENV"
echo "
Hello World!
Here's a
multi-line string.
" >>"$GITHUB_ENV"
echo "EOF" >>"$GITHUB_ENV"
- name: Check the environment variable from the previous step
run: echo $MULTI_LINE_STRING
- name: Set build environment based on Git branch name
if: github.ref == 'refs/heads/demo' || contains(env.APP_NAME, 'demo')
run: echo "BUILD_ENV=demo" >>"$GITHUB_ENV"
- name: Use the GitHub Actions format function to provide some details about Spam
run: |
echo "SPAM_STRING=${{ format(
'Spam is short for {0} and is made from {1} by {2}',
'spiced ham',
'pork shoulder',
'Hormel'
) }}" >>"$GITHUB_ENV"
- name: Run a multi-line shell script block
run: |
echo "
Hello World, from ${{ env.APP_NAME }}!
Add other actions to build,
test, and deploy your project.
"
if [ "$BUILD_ENV" = "demo" ] || ${{ contains(env.APP_NAME, 'demo') }}; then
echo "This is a demo."
elif [ "$BUILD_ENV" ]; then
echo "BUILD_ENV=$BUILD_ENV"
else
echo "There isn't a BUILD_ENV variable set."
fi
echo "Did you know that $SPAM_STRING?"
- uses: actions/setup-python@main
with:
python-version: "3.10"
- name: Run a multi-line Python script block
shell: python
run: |
import os
import sys
version = f"{sys.version_info.major}.{sys.version_info.minor}"
print(f"Hello World, from Python {version} and ${{ env.APP_NAME }}!")
print(f"Did you know that {os.getenv('SPAM_STRING', 'there is a SPAM_STRING')}?")
- name: Run an external shell script
working-directory: ./.github/workflows
run: . github-actions-workflow-demo.sh
- name: Run an external Python script
working-directory: ./.github/workflows
run: python github-actions-workflow-demo.py
name: demo
on:
push:
branches: [master, develop]
pull_request:
workflow_dispatch:
env:
APP_NAME: "GitHub Actions sample workflow with build matrix"
jobs:
matrix:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [macOS-latest, ubuntu-latest, windows-latest]
python-version: ["3.8", "3.9", "3.10"]
silly-word: [foo, bar, baz]
steps:
- uses: actions/checkout@main
- name: Echo a silly word
run: echo ${{ matrix.silly-word }}
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@main
with:
python-version: ${{ matrix.python-version }}
- name: Run a multi-line Python script block
shell: python
run: |
import os
import sys
version = f"{sys.version_info.major}.{sys.version_info.minor}"
print(f"Hello World, from Python {version}, ${{ matrix.os }}, and ${{ matrix.silly-word }}!")GitHub Actions provides output like this:
You can see a demo workflow in br3ndonland/algorithms.
- Steps in a job are sequential by default.
- Jobs are parallelized by default, unless you control the order by using
needs. - Action inputs and outputs: If you're unclear on what you can do with an action, navigate to the GitHub repo for the action and look for a file called action.yml, like this one in actions/checkout. This file is a manifest declaring what the action can do.
- Debugging: If you want more debugging information, add
ACTIONS_STEP_DEBUGto your secrets parameter store.- Key:
ACTIONS_STEP_DEBUG - Value:
true
- Key:
Secrets is an encrypted parameter store (key:value store). The syntax is similar to environment variables.
- GitHub Actions can use secrets, so you don't have to hard-code API keys and other credentials. Secrets are redacted from GitHub Actions logs.
- Each repo has a secrets store at Settings -> Secrets (
https://github.com/org/repo/settings/secrets) - Each organization also has a secrets store that can be used in all the organization's repos.
- Service containers can be set up for test databases and other needs.
- GitHub Actions now allows service containers from private registries.
- Users can create their own actions. For example, container actions can run in Docker containers of your choosing. Action composition can help reduce code duplication.
- GitHub Actions offers a
concurrencykey to control how many workflows are running at one time. - Syntax examples:
- To only allow one concurrent workflow per commit:
concurrency: ci-${{ github.ref }} - To only allow one concurrent workflow per environment, at the job level:
concurrency: ${{ environment.name }}
- To only allow one concurrent workflow per commit:
- Concurrency can be specified at the workflow level, or at the job level.
- It's most useful to specify concurrency at the job level, because outputs from earlier jobs in the workflow can be used (like
concurrency: ${{ needs.setup-job.outputs.ecs-deployment-group-name }}). Unfortunately, job-level concurrency (jobs.<job_id>.concurrency) triggers runs out of order (earlier jobs run after later jobs), which is not very useful. The docs explain, "Note: When concurrency is specified at the job level, order is not guaranteed for jobs or runs that queue within 5 minutes of each other." - Concurrency can also be specified at the workflow level, using the name of the workflow (
concurrency: ${{ github.workflow }}). This avoids triggering runs out of order, but could also result in canceled workflow runs. Only one run can be pending at a time, and each new run will cancel the previous pending run.
- It's most useful to specify concurrency at the job level, because outputs from earlier jobs in the workflow can be used (like
- A common use case for limiting concurrency is in deployments. For example, when deploying Docker containers to AWS ECS Fargate with CodeDeploy, the aws-actions/amazon-ecs-deploy-task-definition step may error with
DeploymentLimitExceededException. The error message will look something like, "The Deployment Group already has an active Deployment." Furthermore, if GitHub Actions workflows are canceled, the ECS Fargate CodeDeploy deployments are not necessarily also canceled. If another GitHub Actions run proceeds, theDeploymentLimitExceededExceptionmay still be seen.- One solution is to add the
force-new-deployment: truesetting to the aws-actions/amazon-ecs-deploy-task-definition step, to ensure that a new deployment is triggered. - It is possible to simply re-run any workflows that failed because of this error.
- It could also be helpful to limit concurrency to a single deployment to avoid errors.
- One solution is to add the
- Overall, GitHub Actions concurrency has many limitations, and may not be useful at this point.
- Note: concurrency is in beta and the syntax may change.
-
The
needs:key helps chain jobs within a workflow together, but not multiple workflows. -
There's a new
workflow_runtrigger that may help to chain workflows together. The GitHub Actions docs explain how to use it to chain workflows:To run a workflow job conditionally based on the result of the previous workflow run, you can use the
jobs.<job_id>.iforjobs.<job_id>.steps[*].ifconditional combined with theconclusionof the previous run. For example:on: workflow_run: workflows: ["Build"] types: [completed] jobs: on-success: runs-on: ubuntu-latest if: github.event.workflow_run.conclusion == 'success' steps: ... on-failure: runs-on: ubuntu-latest if: github.event.workflow_run.conclusion == 'failure' steps: ...
-
GitHub has also introduced some features for reusable workflows, including a new
workflow_calltrigger that allows entire workflows to be called directly. The called workflow must includeon: workflow_call, with inputs that are referenced in the downstream workflow.# greeter.yml name: Reusable workflow example on: workflow_call: inputs: username: description: Username for the workflow required: true type: string repo: description: > Repository to check out. Include a personal access token (PAT) with repo scope for private repos. required: true type: string environment: required: false type: string default: development secrets: personal-access-token: description: GitHub Personal Access Token (PAT) with necessary scopes required: false jobs: greeter: name: Greet the user runs-on: ubuntu-latest steps: uses: actions/checkout@main with: repository: ${{ inputs.repo }} token: ${{ secrets.personal-access-token }} run: | echo 'Hello, ${{ inputs.username }}!' \ 'The repo ${{ inputs.repo }} was checked out.' \ 'This is the ${{ inputs.environment }} environment.'
name: Downstream workflow that uses the reusable workflow on: push: pull_request: workflow_dispatch: jobs: greet_me: uses: org/repo/.github/workflows/greeter.yml@main with: username: Octocat repo: my-private-repo personal-access-token: ${{ secrets.MY_GITHUB_PAT }}
Hello, Octocat! The repo my-private-repo was checked out. This is the development environment.
- Use
continue-on-error: trueon a step to continue the job if the step fails, and useif: steps.<step_id>.outcome == 'success'to trigger downstream steps only if the step succeeds. - Use
continue-on-error: trueon a job to continue to the next job if the current job fails, and useif: jobs.<job_id>.outcome == 'success'to trigger downstream jobs only if the job succeeds.
GitHub Actions offers environments, which allow branch protection rules and secrets to be set individually for each environment (instead of for the repo as a whole). The way they've set up environments is a little confusing, and most of the functionality of environments can be implemented with other features like strategy.matrix or environment variables.
Let's look at a common use case that highlights the limitations of environments: deploying a service to multiple cloud regions. We'll use two environments that map to Git branches, and two cloud regions for each environment.
- In a GitHub repo's settings, add an environment for each combo of Git branch and AWS region. For example, for two Git branches and two AWS regions, the environment names could be:
development-us-east-1development-us-west-2production-us-east-1production-us-west-2
- In each environment's environment protection rules, set the deployment branch to the corresponding Git branch:
development-us-east-1:developdevelopment-us-west-2:developproduction-us-east-1:mainproduction-us-west-2:main
- In the environment secrets for each environment, set a secret with key
AWS_REGIONand value set to the corresponding AWS region. These aren't necessarily sensitive credentials, but GitHub uses secrets for all key-value pairs. Examples:development-us-east-1: secret name:AWS_REGION, secret value:us-east-1development-us-west-2: secret name:AWS_REGION, secret value:us-west-2production-us-east-1: secret name:AWS_REGION, secret value:us-east-1production-us-west-2: secret name:AWS_REGION, secret value:us-west-2
- Reference the environment name within GitHub Actions workflows, optionally limiting workflow concurrency by each environment.
name: multi-region environment example
on:
push:
branches: [develop, main]
workflow_dispatch:
jobs:
setup:
runs-on: ubuntu-latest
outputs:
deploy-env: ${{ steps.set-deploy-env.outputs.deploy-env }}
steps:
- uses: actions/checkout@main
- name: Set deployment environment based on Git branch
id: set-deploy-env
run: |
if ${{ github.ref == 'refs/heads/main' }} || ${{ startsWith(github.ref, 'refs/tags/') }}; then
DEPLOY_ENV="production"
else
DEPLOY_ENV="development"
fi
echo "deploy-env=$DEPLOY_ENV" >>"$GITHUB_OUTPUT"
deploy:
needs: [setup]
runs-on: ubuntu-latest
strategy:
matrix:
aws-region: [us-east-1, us-west-2]
environment:
name: ${{ needs.setup.outputs.deploy-env }}-${{ matrix.aws-region }}
concurrency: ${{ environment.name }}
steps:
- uses: aws-actions/configure-aws-credentials@master
id: aws-login
with:
aws-region: ${{ secrets.AWS_REGION }}
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- run: echo "This job will deploy to the environment named ${{ environment.name }}."By the time you've done all the setup to actually specify the deployment environment and region, you already have the region you need, and the environment isn't particularly useful. For example, in the aws-login step above, the aws-region: ${{ secrets.AWS_REGION }} value could be replaced by aws-region: ${{ matrix.aws-region }}. It would be more helpful to have an environment automatically apply to a Git branch or a workflow, and then be able to reference the configuration values within workflows automatically. It's actually the opposite. You have to specify the environment from within the workflow, and then if the environment protection rules are met, you get access to the corresponding environment secrets.
-
VM info: The GitHub Actions runner provisions virtual machines with similar resources as AWS EC2
t2.largeinstances.- 2-core CPU
- 7 GB of RAM memory
- 14 GB of SSD disk space
-
GitHub Actions is free for open-source repos. Pricing for other repos only kicks in if you exceed the allotted build minutes.
Poetry is a useful tool for Python packaging and dependency management. The following set of tips was originally posted to python-poetry/poetry#366.
Use actions/cache with a variation on their pip cache example to cache Poetry dependencies for faster installation.
- name: Set up Poetry cache for Python dependencies
uses: actions/cache@main
if: startsWith(runner.os, 'Linux')
with:
path: ~/.cache/pypoetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: ${{ runner.os }}-poetry-Installing Poetry via pip can lead to dependency conflicts, and their "custom installer" script (get-poetry.py/install-poetry.py) has been problematic. Instead, Poetry can be installed with pipx. Versions of pipx and Poetry can be pinned to promote reproducible installations.
- name: Install pipx
run: python -m pip install "pipx==0.16.4"
- name: Install Poetry
run: pipx install "poetry==1.1.11"
- name: Install dependencies
run: poetry install --no-interaction- Create a PyPI token.
- Add it to the GitHub Secrets store for the repo (Settings -> Secrets).
- Use the secret in your workflow with
${{ secrets.PYPI_TOKEN }}(secret name isPYPI_TOKENin this example, and username for PyPI tokens is__token__). - Use
poetry publish --buildto build and publish in one step.
- name: Build Python package and publish to PyPI
if: startsWith(github.ref, 'refs/tags/')
run: poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}That's why they call it Poetry. Beautiful.
Expand this details element for an example workflow that uses these tips.
name: ci
on:
pull_request:
push:
branches: [develop, main]
tags:
- "[0-9]+.[0-9]+.[0-9]+*"
workflow_dispatch:
jobs:
ci:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10"]
env:
PIPX_VERSION: "0.16.4"
POETRY_VERSION: "1.1.11"
POETRY_VIRTUALENVS_IN_PROJECT: true
steps:
- uses: actions/checkout@main
- uses: actions/setup-python@main
with:
python-version: ${{ matrix.python-version }}
- name: Set up Poetry cache for Python dependencies
uses: actions/cache@main
if: startsWith(runner.os, 'Linux')
with:
path: ~/.cache/pypoetry
key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
restore-keys: ${{ runner.os }}-poetry-
- name: Install pipx for Python ${{ matrix.python-version }}
run: python -m pip install "pipx==$PIPX_VERSION"
- name: Install Poetry
run: pipx install "poetry==$POETRY_VERSION"
- name: Test Poetry version
run: |
POETRY_VERSION_INSTALLED=$(poetry -V)
echo "The POETRY_VERSION environment variable is set to $POETRY_VERSION."
echo "The installed Poetry version is $POETRY_VERSION_INSTALLED."
case $POETRY_VERSION_INSTALLED in
*$POETRY_VERSION*) echo "Poetry version correct." ;;
*) echo "Poetry version incorrect." && exit 1 ;;
esac
- name: Install dependencies
run: poetry install --no-interaction
- name: Test virtualenv location
run: |
EXPECTED_VIRTUALENV_PATH=${{ github.workspace }}/.venv
INSTALLED_VIRTUALENV_PATH=$(poetry env info --path)
echo "The virtualenv should be at $EXPECTED_VIRTUALENV_PATH."
echo "Poetry is using a virtualenv at $INSTALLED_VIRTUALENV_PATH."
case "$INSTALLED_VIRTUALENV_PATH" in
"$EXPECTED_VIRTUALENV_PATH") echo "Correct Poetry virtualenv." ;;
*) echo "Incorrect Poetry virtualenv." && exit 1 ;;
esac
- name: Run unit tests
run: poetry run pytest tests
- name: Build Python package with latest stable Python version and publish to PyPI
if: matrix.python-version == '3.10' && startsWith(github.ref, 'refs/tags/')
run: |
PACKAGE_VERSION=$(poetry version -s)
GIT_TAG_VERSION=$(echo ${{ github.ref }} | cut -d / -f 3)
echo "The Python package version is $PACKAGE_VERSION."
echo "The Git tag version is $GIT_TAG_VERSION."
if [ "$PACKAGE_VERSION" = "$GIT_TAG_VERSION" ]; then
echo "Versions match."
else
echo "Versions do not match." && exit 1
fi
poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}Dependabot now offers automated version updates, with (preliminary) support for Poetry 🎉. If you have access to the Dependabot beta, set up .github/dependabot.yml as described in the docs:
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"Dependabot will now send you PRs when dependency updates are available. Although package-ecosystem must be set to pip, it will pick up the pyproject.toml and poetry.lock. Check the status of the repo at Insights -> Dependency graph -> Dependabot.
There's no PR merge trigger. To run a workflow when PRs are merged, use pull_request: types: [closed] as the event trigger and only run each job if the GitHub event payload contains merged == 'true'.
name: demo
on:
pull_request:
types: [closed]
jobs:
job1:
runs-on: ubuntu-latest
if: github.event.pull_request.merged == 'true'
steps:
- name: step1
run: echo 'pull request was merged'Triggers have confusing restrictions. Some events will only trigger workflows if the YAML syntax is on the default branch, and some events can only run on the default branch. For example, here is a comparison of repository_dispatch and workflow_dispatch:
repository_dispatch- Where the YAML needs to be:
- Default branch. Docs say "Note: This event will only trigger a workflow run if the workflow file is on the default branch." This means that the YAML file must be committed to the repo's default branch, and the
repository_dispatch:trigger must be in the YAML file.
- Default branch. Docs say "Note: This event will only trigger a workflow run if the workflow file is on the default branch." This means that the YAML file must be committed to the repo's default branch, and the
- Which branches it can run on:
- Default branch only.
- How it's triggered:
- Webhook event through the GitHub API. For example, let's say you use GitHub Actions to build and deploy a Docker container to Azure. It deploys successfully, but later maybe the connection to the database starts failing because of an outdated password or something. You can set up Azure to send a webhook to GitHub, and re-deploy the container.
- Where the YAML needs to be:
workflow_dispatch- Where the YAML needs to be:
- Default branch (though the docs don't say this directly).
- Which branches it can run on:
- Any branch.
- How it's triggered:
- Where the YAML needs to be:
Environment variables can't be set per-trigger. It's difficult to set environment variables per-trigger, such as based on which branch was checked out. There's a top-level env: key, but it doesn't allow expressions or separate steps:.
env: # Can't do this
- name: Set build environment to production if master branch is checked out
if: contains(github.ref, 'master')
run: echo "BUILD_ENV=production" >>"$GITHUB_ENV"
- name: Set build environment to development if develop branch is checked out
if: contains(github.ref, 'develop')
run: echo "BUILD_ENV=development" >>"$GITHUB_ENV"
- name: Set build environment to test otherwise
if: ${{ !contains(github.ref, 'master') || !contains(github.ref, 'develop') }}
run: echo "BUILD_ENV=test" >>"$GITHUB_ENV"Furthermore, environment variables set to $GITHUB_ENV within a job are scoped to that job.
The solution is to use outputs instead of environment variables. Set outputs from one job and read them in downstream jobs. Note that step outputs must also be set as job outputs in order to be passed to other workflows.
-
If a PR has merge conflicts, GitHub Actions workflows may not run at all. See this GitHub community thread.
-
Try using equality operators (
==and!=) to check out the PRHEADcommit, instead of the default (the merge commit), as described in the actions/checkout README:- uses: actions/checkout@main if: ${{ github.event_name != 'pull_request' }} - uses: actions/checkout@main if: ${{ github.event_name == 'pull_request' }} with: ref: ${{ github.event.pull_request.head.sha }}
-
If checking out the
HEADcommit doesn't work, you may need to resolve merge conflicts to continue.
-
See the context and expression syntax docs
-
In which context can I use which things?
- Where can I define
env:? - Where can I define
defaults:? - Where can I add
if:? - Where do I need
${{ }}for expressions? The docs explain thatif:conditionals are automatically evaluated as expressions, but it's not always clear in other fields.
- Where can I define
-
There are syntactic subtleties, such as the requirement for single quotes in some YAML contexts:
jobs: job1: runs-on: ubuntu-latest steps: # this works - name: Set build environment to production if: github.ref == 'refs/heads/master' run: echo "BUILD_ENV=production" >>"$GITHUB_ENV" # this doesn't because of "" in the if: - name: Set build environment to production if: github.ref == "refs/heads/master" run: echo "BUILD_ENV=production" >>"$GITHUB_ENV"
-
Or the requirement for quoting in other contexts, such as for Python 3.10:
jobs: job1: runs-on: ubuntu-latest strategy: matrix: # 3.10 without quotes will be parsed as 3.1 python-version: [3.8, 3.9, "3.10"] steps: - uses: actions/checkout@main - uses: actions/setup-python@main with: python-version: ${{ matrix.python-version }} - run: echo "Python version is ${{ matrix.python-version }}"
-
There's no concept of
if/elif/else. -
Object filters seem useful, but there's no explanation for how to set up object filters within workflows.
-
There are sometimes several context keys with similar values , like
github.base_refvs.github.ref_namevs.github.ref:github.base_refandgithub.ref_namereturn just the branch name, likemain.github.refreturns the full Git ref, likerefs/heads/main.
- Overview
- Workflow triggers (the
on:section of YAML workflow files): - Workflow syntax
- GitHub docs: Actions - Commands (special commands you can send to the runner within workflows, like
echo "COLOR=green" >>"$GITHUB_ENV") - GitHub docs: Actions - Contexts (metadata like
${{ github.repository }}) - GitHub docs: Actions - Expressions (syntax in double curly braces like
${{ format() }}) - GitHub docs: Actions - Workflow syntax reference
- GitHub docs: Actions - Commands (special commands you can send to the runner within workflows, like
- GitHub Actions + GitHub Container Registry
- Actions
- GitHub Actions Marketplace
- actions/runner: the runner for GitHub Actions
- sdras/awesome-actions
- A Gist is actually a repository.
- To clone the Gist locally:
git clone git@gist.github.com:f9c753eb27381f97336aa21b8d932be6.git github-actions
- No subdirectories are allowed.
- To add images to a Markdown file in a Gist:
- Commit the image file (in the same directory, no sub-directories)
- Push the change to GitHub with
git push - Click on the "raw" button next to the image
- Copy the URL
- Add the URL to an image tag in the Markdown file.
Hello,
Please see https://gist.github.com/sammcj/700787ee00b7aa63d4fc4e536d45b12f (as gists don't support PRs).
Updates:
set-outputandset-envcommands.