Speed up your multistage builds in GitHub Actions

Caching multi-stage builds in GA.md

Caching multi-stage builds in GitHub Actions

Caching Docker builds in GitHub Actions is an excellent article by @dtinth which analyses various strategies for speeding up builds in GitHub Actions. The upshot of the article is a fairly decisive conclusion that the best two ways to improve build times are:

1. Build images via a standard docker build command, while using GitHub Packages' Docker registry as a cache = Longer initial build but fastest re-build times.

2. Build your images via docker integrated BuildKit (DOCKER_BUILDKIT=1 docker build), while using a local registry and actions/cache to persist build caches = Fastest initial build but slightly longer re-build times.

The problem

Unfortunately, when trying to implement these solutions in my own project, I discovered that both approaches fall short when it comes to multi-stage builds.

The problem is that inlined build cache only includes layers directly involved in the creation of the image. It excludes caches from various stages in my Dockerfile which are copied into the final image using

COPY --from=...

Using the above strategies did improve my image re-building time. It has gone down from ~ 3m 15s to about 1m 50s. However, this improvement was not as much as I would have liked...

The solution

Luckily, the guys behind docker integrated BuildKit have a standalone tool for building docker images. The tool comes in two parts: buildkitd - a build daemon, and buildctl - a controller for the build daemon.

The features that I was most interested in, is the ability of BuildKit controller (buildkitctl) to export/import a full set of build caches, including caches for all multi-stage layers. Combining this ability with actions/cache gave me a nearly perfect solution for speeding up multi-stage docker builds in Github Actions.

Final time to re-build the image has now been reduced from ~ 3m 15s to ~ 38s!

---

PS: Interestingly, BuildKit daemon (buildkitd) can run either locally or remotely. This opens up the possibility of hosting a standalone build process on your own infrastructure, which might further improve building time.

example-workflow.yml

name: CI pipeline

on:
  push:
    branches: [master]
  pull_request:

defaults:
  run:
    shell: bash

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      DOCKER_IMAGE: ci/${{ github.job }}
      BUILD_CACHE: /home/runner/.docker/buildkit
    steps:
      - uses: actions/checkout@v2

      - uses: actions/cache@v1
        with:
          path: ${{ env.BUILD_CACHE }}
          key: ${{ hashFiles('Dockerfile') }}

      - name: Install the latest buildkit release
        run: |
          BUILDKIT_URL="$(curl -sL https://api.github.com/repos/moby/buildkit/releases \
            | jq -r 'map(select((.name|startswith("v")) and (.name|contains("rc")|not)))|sort_by(.published_at)[-1].assets[]|select(.name|endswith(".linux-amd64.tar.gz")).browser_download_url')"
          curl -L "${BUILDKIT_URL}" | sudo tar -xz -C /usr/local

      - name: Start buildkit daemon
        run: |
          sudo --non-interactive --shell <<END_SUDO
            install -d -m 0750 -o root -g docker /run/buildkit
            buildkitd &
            while ! test -S /run/buildkit/buildkitd.sock; do sleep 0.1; done
            chgrp docker /run/buildkit/buildkitd.sock
          END_SUDO

      - name: Build docker image
        run: |
          buildctl build \
            --frontend=dockerfile.v0 --local dockerfile=. --local context=. \
            --export-cache type=local,dest=${BUILD_CACHE},mode=max \
            --import-cache type=local,src=${BUILD_CACHE} \
            --output type=docker,name=${DOCKER_IMAGE} | docker load
          echo "Cache size: $(du -sh ${BUILD_CACHE})"

      - name: Launch a container based on the new image (example)
        run: ./bin/docker run --rm ${DOCKER_IMAGE} ...

You can also set --build-arg BUILDKIT_INLINE_CACHE=1 during the docker build this will add the cache metadata to the built image which when pushed an pulled from a registry will allow you to cache the layers of the multi-stage build.

https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources

Worked for me taking a hash of the directories of interest as key of the cache and restoring the previous cache if the key change.

jobs:
  test:
    runs-on: ubuntu-latest
     env:
       BUILD_CACHE: /home/runner/.docker/buildkit

     steps:
       - name: Checkout code
         uses: actions/checkout@v2
       - name: Compute cache key
         run: echo "CACHE_KEY=$(git ls-files -s dir1 dir2 | git hash-object --stdin)" >> $GITHUB_ENV
       - name: Cache docker layers
         uses: actions/cache@v1
         with:
           path: ${{ env.BUILD_CACHE }}
           key: cache-${{ env.CACHE_KEY }}
           restore-keys: |
             cache-

...

Thanks @UrsaDK

There is a minor error in sorting buildkit releases by version. It sorts it alphabetically rather than by numbers. Also, it takes into account release candidates. So, it could find 0.9-rc before 0.10. To fix that we could sort by the publishing date, and filter out all the release candidates:

BUILDKIT_URL="$(curl -sL https://api.github.com/repos/moby/buildkit/releases \
            | jq -r 'map(select((.name|startswith("v")) and (.name|contains("rc")|not)))|sort_by(.published_at)[-1].assets[]|select(.name|endswith(".linux-amd64.tar.gz")).browser_download_url')"

Nice one! Thanks for this @dantonyuk 👍 I've updated the gist to include your fix. 🙂