Skip to content

Instantly share code, notes, and snippets.

@vanstee

vanstee/INFO.md

Last active May 27, 2020 15:06
Show Gist options
  • Save vanstee/85ae640d5b97a7124fc4d2d91e80d7b3 to your computer and use it in GitHub Desktop.
Save vanstee/85ae640d5b97a7124fc4d2d91e80d7b3 to your computer and use it in GitHub Desktop.
Download ZIP
Docker COPY busting cache on CentOS
Raw
INFO.md

Docker COPY busting cache on CentOS

We build a large number of images in CI and rely on images built with buildkit that contain inline cache. Recently, after switching from using buildctl build directly to using docker build with DOCKER_BUILDKIT=1 exported and the BUILDKIT_INLINE_CACHE=1 build arg, we've noticed that cache works as expected only up until a COPY instruction, as shown with steps to reproduce below.

Build on OS X and push to Docker Hub

Build an image with inline cache, push it to Docker Hub, clear out everything locally, and show that rebuilding from cache works correctly.

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., 0.0.0+unknown)

Server:
 Containers: 25
  Running: 24
  Paused: 0
  Stopped: 1
 Images: 14
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.945GiB
 Name: docker-desktop
 ID: MSI5:RISV:MJPQ:UCVR:I5VT:HXAA:3KSL:GMYJ:PTPE:XCOW:LX4I:J2IC
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 171
  Goroutines: 147
  System Time: 2020-05-27T14:58:47.3048602Z
  EventsListeners: 4
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  localhost:5000
  0.0.0.0:5000
  192.168.1.72:5000
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine


$ cat <<EOF > Dockerfile
FROM alpine
RUN echo foo
COPY bar bar
RUN echo baz
EOF
$ echo bar > bar
$ DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --tag vanstee/test .         
[+] Building 3.4s (10/10) FINISHED                       
 => [internal] load .dockerignore                                                                            0.1s 
 => => transferring context: 2B                                                                              0.0s
 => [internal] load build definition from Dockerfile                                                         0.2s
 => => transferring dockerfile: 91B                                                                          0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                             0.5s
 => [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb  1.0s
 => => resolve docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb  0.0s
 => => sha256:39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01 528B / 528B                   0.0s
 => => sha256:f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a 1.51kB / 1.51kB               0.0s
 => => sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 2.81MB / 2.81MB               0.4s
 => => sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb9a54 1.64kB / 1.64kB               0.0s
 => => extracting sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08                    0.3s
 => [internal] load build context                                                                            0.0s
 => => transferring context: 34B                                                                             0.0s
 => [2/4] RUN echo foo                                                                                       0.7s
 => [3/4] COPY bar bar                                                                                       0.1s
 => [4/4] RUN echo baz                                                                                       0.6s
 => exporting to image                                                                                       0.0s
 => => exporting layers                                                                                      0.0s
 => => writing image sha256:7334553d104b0b485f1dc613d4f5005917ec93864fb64f2f2fc7371851f764d0                 0.0s
 => => naming to docker.io/vanstee/test                                                                      0.0s
 => exporting cache                                                                                          0.0s
 => => preparing build cache for export                                                                      0.0s
$ docker push vanstee/test                                                                                        
The push refers to repository [docker.io/vanstee/test]                                                            
5f70bf18a086: Layer already exists 
9a178b6109c6: Pushed 
c338759da1a5: Pushed 
3e207b409db3: Layer already exists 
latest: digest: sha256:151842699becdaaaa461c7c64d4b6ac38e88e8799263de5d7638e38c52165327 size: 1148
$ docker system prune -a -f > /dev/null
$ DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from vanstee/test --tag vanstee/test .
[+] Building 2.7s (11/11) FINISHED                                                                                
 => [internal] load build definition from Dockerfile                                                         0.1s
 => => transferring dockerfile: 91B                                                                          0.0s
 => [internal] load .dockerignore                                                                            0.1s
 => => transferring context: 2B                                                                              0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                             0.5s
 => importing cache manifest from vanstee/test                                                               0.4s
 => [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb  0.0s
 => [internal] load build context                                                                            0.0s
 => => transferring context: 34B                                                                             0.0s
 => CACHED [2/4] RUN echo foo                                                                                0.0s
 => CACHED [3/4] COPY bar bar                                                                                0.0s
 => CACHED [4/4] RUN echo baz                                                                                1.6s
 => => pulling sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08                       0.3s
 => => pulling sha256:a6a654dae9dc15ecd273df6c7c8149e60f8a2f2f2374a5e1ea0b7316359c949d                       0.3s
 => => pulling sha256:868e8a0ced0ebe5a970c6767909c0e06189a2f0ab908bf657d04fc58da06495d                       0.3s
 => => pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1                       0.1s
 => exporting to image                                                                                       0.0s
 => => exporting layers                                                                                      0.0s
 => => writing image sha256:7334553d104b0b485f1dc613d4f5005917ec93864fb64f2f2fc7371851f764d0                 0.0s
 => => naming to docker.io/vanstee/test                                                                      0.0s
 => exporting cache                                                                                          0.0s
 => => preparing build cache for export                                                                      0.0s

Build on Ubuntu using cache image from OS X

Build the same image using the cache from the previoulsy built image and show that everything works fine.

# docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-58-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 985.5MiB
 Name: vagrant
 ID: QDUF:7BKT:LZVN:NQDV:R2ZJ:NN7D:NN7H:2OLM:6EN2:2VHV:BJ7C:WA5N
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

# cat <<EOF > Dockerfile
FROM alpine
RUN echo foo
COPY bar bar
RUN echo baz
EOF
# echo bar > bar
# DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from vanstee/test --tag vanstee/test .                                
[+] Building 5.2s (11/11) FINISHED                       
 => [internal] load build definition from Dockerfile                                                         0.1s
 => => transferring dockerfile: 88B                                                                          0.0s
 => [internal] load .dockerignore                                                                            0.1s
 => => transferring context: 2B                                                                              0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                             3.1s
 => importing cache manifest from vanstee/test                                                               0.6s
 => [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb  0.0s
 => [internal] load build context                                                                            0.0s
 => => transferring context: 34B                                                                             0.0s
 => CACHED [2/4] RUN echo foo                                                                                0.0s
 => CACHED [3/4] COPY bar bar                                                                                0.0s
 => CACHED [4/4] RUN echo baz                                                                                1.3s
 => => pulling sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08                       0.4s
 => => pulling sha256:868e8a0ced0ebe5a970c6767909c0e06189a2f0ab908bf657d04fc58da06495d                       0.1s
 => => pulling sha256:a6a654dae9dc15ecd273df6c7c8149e60f8a2f2f2374a5e1ea0b7316359c949d                       0.2s
 => => pulling sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1                       0.1s
 => exporting to image                                                                                       0.0s
 => => exporting layers                                                                                      0.0s
 => => writing image sha256:7334553d104b0b485f1dc613d4f5005917ec93864fb64f2f2fc7371851f764d0                 0.0s
 => => naming to docker.io/vanstee/test                                                                      0.0s
 => exporting cache                                                                                          0.0s
 => => preparing build cache for export                                                                      0.0s

Build on CentOS using cache image from OS X

Build the same image using the cache from the previoulsy built image and show that only the first instruction is cached.

# docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
# docker info
Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1127.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 487MiB
 Name: localhost.localdomain
 ID: ZITA:KMKP:5JI7:OQPA:IHJ3:RKRT:IKMR:GOVV:77AO:UIH5:KYSF:3RXQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

# cat <<EOF > Dockerfile
FROM alpine
RUN echo foo
COPY bar bar
RUN echo baz
EOF
# echo bar > bar
# DOCKER_BUILDKIT=1 docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from vanstee/test --tag vanstee/test .
[+] Building 2.8s (11/11) FINISHED                                                                                
 => [internal] load build definition from Dockerfile                                                         0.0s
 => => transferring dockerfile: 146B                                                                         0.0s
 => [internal] load .dockerignore                                                                            0.0s
 => => transferring context: 2B                                                                              0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                             1.0s
 => importing cache manifest from vanstee/test                                                               0.4s
 => [1/4] FROM docker.io/library/alpine@sha256:9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb  0.0s
 => [internal] load build context                                                                            0.0s
 => => transferring context: 92B                                                                             0.0s
 => CACHED [2/4] RUN echo foo                                                                                0.7s
 => => pulling sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08                       0.2s
 => => pulling sha256:868e8a0ced0ebe5a970c6767909c0e06189a2f0ab908bf657d04fc58da06495d                       0.1s
 => [3/4] COPY bar bar                                                                                       0.0s
 => [4/4] RUN echo baz                                                                                       0.5s
 => exporting to image                                                                                       0.0s
 => => exporting layers                                                                                      0.0s
 => => writing image sha256:61a1aa8f6d5c8291e80f47aceacee8bd9322c815b0160e3bc22ba18f735e3085                 0.0s
 => => naming to docker.io/vanstee/test                                                                      0.0s
 => exporting cache                                                                                          0.0s
 => => preparing build cache for export                                                                      0.0s

Notice that only the first instruction is cached, and once the COPY instruction is run the cache is no longer used. We were able to replicate this every time a COPY instruction was introduced when building on CentOS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment