Skip to content

Instantly share code, notes, and snippets.

@kanzure
Created February 13, 2016 17:12
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save kanzure/449fe6849516f73d70e8 to your computer and use it in GitHub Desktop.
Unexpected docker build cache miss after successful --no-cache
git checkout master # setup Dockerfile
docker-compose build # uses cache (expected)
git checkout HEAD^1 # removes some line from Dockerfile
docker-compose build # uses cache (expected)
git checkout master # adds back same line to Dockerfile
docker-compose build --no-cache # does not use input from the cache (expected)
git checkout HEAD^1 # removes same line from Dockerfile
docker-compose build # has unexpected cache miss

possibly related? moby/moby#3199

@kanzure
Copy link
Author

kanzure commented Feb 13, 2016

Wild guess: Recency is not enough

"Use the most recent image from the build cache" is implemented here: https://github.com/docker/docker/blob/415dd8688618ac2b3872c6a5b5af3afcef2c3e10/daemon/daemon.go#L1363

"Use the most recent image from the build cache" will fail because the last known good build may have used a different image. It's not enough to sort by build cache item recency. Cache miss event should cause the builder to backtrack and try the next most recent available matching image, until it exhausts all available image options, at which point it should backtrack to the previous Dockerfile step and repeat the exhaustive search.

"Use the most recent image from the build cache" will fail after using --no-cache because "last known most recently matching image" will naturally not have any relevant child images in the build cache. The "last known most recently matching image" will be whatever image was recently created during --no-cache, and therefore have none of the children from previous cached builds. Meanwhile the build cache is still present and available and has useful things inside.

Confusion intensifies

.... turns out that I am not using "--no-cache" anyway, but still seeing lots of cache misses.

Tool for investigating the status of the docker build cache?

There should be more tools to display the status of the docker build cache. There should also be a tool to do a "dry run" of a docker build, which would display information about the docker build cache, without actually manipulating the docker build cache and without running an actual build.

Other related

"faster cache miss detection"
moby/moby#16821

"output reason for docker cache invalidation"
moby/moby#9294

@kanzure
Copy link
Author

kanzure commented Feb 14, 2016

Reported this upstream to docker, moby/moby#20304

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment