Don't use golden images. Do this instead.

golden_images.md

tl;dr: If you run apt-get install reproducibly, there's no reason to use "golden images".

Background (about me)

I've now run into the "how do we manage and update a base operating system" problem at three different roles over the course of many years. At each role, my colleagues and I landed on a reproducible installations using apt . Fair warning: the rest of this post will be apt flavored, although I hope that the general lessons will be useful for any operating system.

The most advanced incarnation of this system used Bazel as part of a monorepo to provide automated upgrades for hundreds of individual services. In that incarnation, it was possible to build new container images that remediated a given CVE org-wide in a single commit, which was very useful for me as a security engineer.

What are golden images?

A golden image is an image that is (a) periodically published for consumption (b) as a base image for another build (c) containing a set of "basic" utilities.

The typical paradigm for using a golden image is to reference the image by tag or digest (FROM), install any requisite OS-level dependencies (e.g.RUN apt-get update && apt-get -y install), and then install the application together with any application-level dependencies.

The whole workflow is therefore:

1. Build job A builds the golden image, runs a set of tests, and publishes to a registry
2. Build job B (or an unlucky human being) updates the build definitions for every image that depends on a now-obsolete golden image.
3. Build job C builds the application and runs tests.

You can also obviate build job B by just referencing the "latest" golden image in build job C. This makes job C non-reproducible, and means that a single change in job A can break numerous downstream builds all-at-once.

All of this is way too much hassle, so let's get rid of golden images. Instead, we'll make all of those steps into a single job, and avoid publishing any "intermediate" or "base" images.

What is a base OS anyway?

A container image is a list of tarballs (layers), plus a bit of metadata describing how to present them to a container runtime. In this context, a "base OS" usually refers to the bottom layer(s) of a container image, and often contains the foundational libraries needed for applications to work (e.g. libc), a package manager (e.g. apt or yum), and basic tools (e.g. bash, ls, cat).

Base OSes are highly optional. Workloads that are static binaries can operate perfectly fine without any other files at all, let alone a "full" operating system. And plenty of dynamically linked applications have no use for package managers or shell utilities, which is why the distroless project publishes some extremely minified images that are designed for use as golden images.

How do I get a base OS?

For the purposes of this post, let's assume we want a base OS because we want to use apt-get.

To get a base image, there are at least three perfectly good approaches. Each one has fewer middlemen than the last:

1. pull and cache Debian or Ubuntu from the Docker hub
2. download and cache a container rootfs from Canonical directly
3. build the image from its constituent packages using something like debootstrap (a well-supported project), or debootstrap-py (a completely unsupported project I threw together over a weekend)

So long as the image is the right major version (e.g. Ubuntu Focal or Debian 11), we don't have to worry about it being a bit out of date. We'll fix that any out-of-date packages later on.

Reproducibility

When a service builds atop a golden image, their Dockerfile often looks like this:

FROM golden:version
RUN apt-get update && \
      apt-get install --no-install-recommends -y ...

In other words, they start by installing a custom set of packages from an external source. Those packages are completely separate from the golden image build. This introduces two wrinkles, which readers may have already noticed:

1. the OS package manager is going to install the requested packages, plus all of their dependencies, plus their dependencies, going down the dependency tree until no more packages are needed (some folks call this the "transitive closure" of dependencies).
2. the OS package manager is acting on external data that changes over time (i.e. apt-get update), so it may make different decisions over time.

The net effect is that we have no clue what packages are actually going to get installed and no understanding of their versions. Our build can change without any underlying code change.

To reiterate, this means:

1. a build on my laptop might be very different from a build on my colleague's laptop
2. a build today might be very different from a build yesterday, or last year
3. we cannot assign a meaningful version number (e.g. a git SHA) to any given build
4. CI against our image is unstable, because each CI run might test a different image
5. if a bad change is introduced in the upstream package repositories, we cannot "revert" it because there is no commit in our VCS to revert

This is bad. The solution is to compute the transitive closure of packages-to-be-installed before the build, output the results to a "lock" file, check the lockfile into your repository, and then use the lockfile during the build. This is what language-specific package managers (e.g. cargo, pip, npm) do, but unfortunately legacy OS package managers like apt and yum haven't caught on.

How do I make the lockfile?

For any package manager (not just apt), the general technique for generating a lockfile is:

1. do the installation without a lockfile
2. dump a list of installed packages

apt-get has the ability to simulate the installation in Step 1, which means that we can skip the heavy lifting of installing the packages. Instead, we just run apt-get -s install <package names> | grep '^Inst ' in order to determine what would be installed. That list, when combined with the list of packages already on the system, becomes our lockfile.

How do I install from the lockfile?

It's slightly difficult. The best solution I've seen puts all of the packages-to-be-installed into a directory, marks that as an apt repository source, uses dpkg --set-selections to indicate which packages should be installed, and then apt-get dselect-upgrade to effect the installation.

The distroless project takes a different approach, which is to skip apt entirely. Instead, they take individual .deb packages, un-archive them, and extract the files directly into the disk image. The problems with this approach are:

1. it is quite possible to install a package without its dependencies
2. apt has a facility to run pre- and post-install scripts, which is skipped
3. apt doesn't recognize the package as installed.

The major benefit of the distroless approach is that it can be done by a very simple program that doesn't run apt-get at all. Unarchiving some files is a lot easier than setting up a Linux container and running apt-get, and can be done in any environment with no additional tooling (e.g. on a Mac laptop). It's also usually a lot faster.

Auto-upgradability

Once a lockfile is present in a repository, dependabot and dependabot-like solutions become available for keeping packages up-to-date. This can be as simple as a cronjob that periodically re-runs the lockfile generator job, and commits the new results.

The net effect is that security upgrades become a three step process:

1. bump version numbers in a lockfile
2. build and test the new image
3. deploy the image

Projects with strong testing can perform these steps entirely automatically. Security engineers love this, and they use the buzzword "shift-left", referring to the fact that security is managed early-on in the build process.

Do I even need customization?

Some people ask:

Most of my builds don't install OS-level packages at all. Why bother with the complexity of providing custom OS images?

This is assumption is frequently true, in an 80-20 way. 80% of builds would be perfectly happy with no OS-level packages, especially in a golang shop where the typical application is a single go binary with essentially no external dependencies.

The problem is that the remaining 20% of use-cases are ill-served, and these 20% are often things like critical infrastructure services that run third party OSS software (e.g. a load balancer, or a data analytics platform), or vendor-produced software.

Can I just use a kitchen-sink image?

You can, but the maintenance burden becomes pretty annoying.

I was once the guy who was responsible for curating the list of packages that would get installed on every machine in my organization's fleet. I don't recommend it.

Let's say you're a Java shop, so you install some useful tool to debug JVMs. This now gets shipped (along with all of its dependencies, including a JVM) to every image in the fleet, even the ones that have no interest in Java. So now every container is flagged as "needing" a security update whenever a there's a new JVM released, and all of the images carry megabytes (eventually gigabytes, as the kitchen sink gradually grows) of deadweight.

Another issue is implicit dependencies. It's much easier to install a common tool (e.g. git) in every image than it is to remove that tool in the future, because applications may have started implicitly depending on git without the application owner declaring the dependency (or even knowing about it themselves). So today's kitchen sink becomes tomorrow's tech-debt.

Finding broken dependencies

If you ever find yourself trying to remediate a situation where someone used a "kitchen-sink" image in the past, you'll likely need to audit all potential call sites to ensure that folks aren't (e.g.) shelling out to git before removing git.

One major exception is with shared libraries. ELF libraries and binaries contain metadata fields that indicate what libraries they depend on. It's totally possible to write a test that uses ldd (or objdump or readelf) to check whether a given container image is shipping all of the requisite dependencies to run a given binary¹, a technique that I first saw with dpkg-shlibdeps. The net effect is that you can eliminate one way to ship a broken image to prod.

What about SBOMs?

In theory, a fully specified and reproducible build process would do away with the need for SBOMs, because the build tool captures the entire software supply chain.

In practice, SBOMs are still a useful and standardized way to take all of the things that your build tool already knows, and export it to a central location for consumption by other tools. I really like the duo of syft + grype for this purpose.

Standing on the shoulders of giants

I'll close this post by mentioning just how grateful we should all be for our OS maintainers, who package upstream software releases and, crucially, curate a trusted channel of security updates. This is an incredible service that we receive every time we run apt-get update.

What about NixOS?

NixOS is an operating system distribution that serves our primary purpose quite well. I haven't personally used it, for two reasons:

1. the Nix filesystem layout differs significantly from a "legacy" OS based on something like apt or yum, adding more complexity to the process of upgrading existing images.
2. NixOS releases a new stable channel every six months, and they support each stable release for seven months. This leaves folks using their stable release 1 month to upgrade to the newer release before losing security support. By contrast, OSes like Ubuntu issue LTS releases every 2 years, and support each stable release for 5 years. This gives users a full three years to work out their upgrade strategy before losing support.

Footnotes

1. Except if you're using dlopen. ↩