Optimizing Rust container builds

README.md

Optimizing Rust container builds

I'm a Rust newbie, and one of the things that I've found frustrating is that the default docker build experience is extremely slow. As it downloads crates, then dependencies, then finally my app - I often get distracted, start doing something else, then come back several minutes later and forget what I was doing

Recently, I had the idea to make it a little better by combining multistage builds with some of the amazing features from BuildKit. Specifically, cache mounts, which let a build container cache directories for compilers & package managers. Here's a quick annotated before & after from a real app I encountered.

Before

This is a standard enough multistage Dockerfile. Nothing seemingly terrible or great here - just a normal build stage, and a smaller runtime stage.

FROM rust:1.55 AS build

WORKDIR /app

# The app has 2 parts: an application named "api", and a lib named "game"

# Copy the sources
COPY ./api ./api
COPY ./game ./game

# Build the app
WORKDIR /app/api
RUN cargo build --release

# Use a slim Dockerfile with just our app to publish
FROM debian:buster-slim AS app

COPY --from=build /app/target/release/my-app /

CMD ["/my-app"]

This corresponds to the following build times

# Let's pre-pull the bases so we don't unnecessarily penalize the first build
docker pull rust:1.55
docker pull debian:buster-slim

# First build from scratch
time docker build .

real    5m43.506s
user    0m1.239s
sys     0m0.872s

# Change a file in api/src, and build again
time docker build .

real    5m44.731s
user    0m1.199s
sys     0m0.938s

Wow, 5 minutes. Yes, I'm probably doing cargo build outside of Docker and the real effects aren't this drastic, but this is an eternity for my short attention span. This is our baseline - let's see if we can improve it.

After

Here we're going to keep multistage builds, but we'll make a few changes:

1. Split layers so that we cache compiled dependencies. Turns out this is harder in Rust than other languages.
2. Use BuildKit + cache mounts. This will save us some download time when we have to rebuild dependencies

# syntax=docker/dockerfile:1.3-labs

# The above line is so we can use can use heredocs in Dockerfiles. No more && and \!
# https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/

FROM rust:1.55 AS build

# Capture dependencies
COPY Cargo.toml Cargo.lock /app/

# We create a new lib and then use our own Cargo.toml
RUN cargo new --lib /app/game
COPY game/Cargo.toml /app/game/

# We do the same for our app
RUN cargo new /app/api
COPY api/Cargo.toml /app/api/

# This step compiles only our dependencies and saves them in a layer. This is the most impactful time savings
# Note the use of --mount=type=cache. On subsequent runs, we'll have the crates already downloaded
WORKDIR /app/api
RUN --mount=type=cache,target=/usr/local/cargo/registry cargo build --release

# Copy our sources
COPY ./api /app/api
COPY ./game /app/game

# A bit of magic here!
# * We're mounting that cache again to use during the build, otherwise it's not present and we'll have to download those again - bad!
# * EOF syntax is neat but not without its drawbacks. We need to `set -e`, otherwise a failing command is going to continue on
# * Rust here is a bit fiddly, so we'll touch the files (even though we copied over them) to force a new build
RUN --mount=type=cache,target=/usr/local/cargo/registry <<EOF
  set -e
  # update timestamps to force a new build
  touch /app/game/src/lib.rs /app/api/src/main.rs
  cargo build --release
EOF

CMD ["/app/target/release/my-app"]

# Again, our final image is the same - a slim base and just our app
FROM debian:buster-slim AS app
COPY --from=build /app/target/release/my-app /my-app
CMD ["/my-app"]

And the big test - did it help at all? Let's see

# We have rust / debian pulled from before

# We need to use BuildKit for these features, so let's turn that on
export DOCKER_BUILDKIT=1

# Build from scratch!
time docker build .

real    5m51.538s
user    0m1.209s
sys     0m0.933s

# The big moment - change a file in src and rebuild
time docker build .

real    0m36.053s
user    0m0.148s
sys     0m0.145s

Great success! Container build times dropped from 5m44s to 0m36s!

Thanks for this guide - kindly change the link for [cache mounts] from https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#run---mounttypecache to https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#run---mounttypecache

In later versions of docker/buildkit anyway, this guide's exact build command does not appear to work. (or at least it didn't work for me)

Specifically, due to using a cache mount for the target folder, the binary produced from the build is left in the cache as well, meaning it is inaccessible to later steps (eg. RUN ls target).

To fix, you must move the produced binary outside of the cached target folder prior to that step's completion, as seen here: ectobit.com/blog/rust-container-image-buildkit-buildx

Example:

RUN --mount=type=cache,target=/usr/local/cargo/registry,id=${TARGETPLATFORM} --mount=type=cache,target=/root/target,id=${TARGETPLATFORM} \
    cargo build --release && \
    cargo strip && \
    mv /root/target/release/<your-crate-name> /root      <-- THIS IS THE MISSING STEP

P.S. I've seen like three versions of this answer neglect to mention the necessary mv target/my-app folder-outside-of-cache/my-app, so my guess is either that it's a difference in configuration somewhere, or that the issue is due to a recent change to the way cache-mounts work in docker/buildkit. (either way, would be nice if someone could find documentation on the behavior change)

How do you set this up if you only have one crate? Does this work with GitHub Actions or other build tools?

Is this correct?

# syntax=docker/dockerfile:1.3-labs

ARG TARGET_PLATFORM=aarch64-apple-darwin
ARG CRATE_NAME=rust-api

FROM rust:1.65 AS build

ARG TARGET_PLATFORM
ARG CRATE_NAME

RUN cargo init --name ${CRATE_NAME}
COPY Cargo.toml .
COPY Cargo.lock .

RUN --mount=type=cache,target=/usr/local/cargo/registry,id=${TARGET_PLATFORM} --mount=type=cache,target=/target,id=${TARGET_PLATFORM} \
    cargo build --release && \
    mv /target/release/${CRATE_NAME} . 

COPY . .

RUN --mount=type=cache,target=/usr/local/cargo/registry <<EOF
  set -e
  touch /src/main.rs
  cargo build --release
EOF

CMD ["/target/release/${CRATE_NAME}"]

FROM debian:buster-slim AS app

ARG CRATE_NAME

COPY --from=build /target/release/${CRATE_NAME} .
COPY --from=build ./.env ./.env

EXPOSE 8080
CMD ["/${CRATE_NAME}"]

I doubt this would work for CI based build, since the mount part assumes you have something persisted already in a specific building "machine"...

I like go mod download， It's friendly to docker build.

Why the target is /usr/local/cargo/registry? Isn't it ~/.cargo?

Depends on CARGO_HOME env var.