Cleaning up Docker images on Jenkins build machines
We use Jenkins to build production ready Docker images here at TotallyMoney. The setup works really well, but we encounter regular problems with high disk usage from Docker. This is primarily because old, untagged images aren't cleaned up either during a new image build or on a timed basis. This is safe of course, particularly for data-only containers, but for our purposes we only need the image to exist for the duration of a build. Once the image is built, it's pushed to the TotallyMoney private registry and lives there (on S3) happily ever after.
The solution I came up with is to just run a bunch of Docker cleanup commands at midnight every day on the Jenkins build machines. It's not particularly elegant, but for our purposes it does what we need, which is to free disk space.
This article was written for Docker 1.7.0, but will most likely work with newer/older versions.
Note: This setup will stop and remove all images and dangling containers (every day if you set up a Cron job), including data-only containers. This won't be a good solution for everyone, but is good for transient containers that are stored elsewhere after they're built, like ours are.
Cleanup steps
- Stop all containers with
docker stop $(docker ps -q) - Remove all stopped containers (they're one-time-use for us anyway) with
docker rm $(docker ps -a -q) - Remove all dangling images (tagged as
<none>) withdocker rmi $(docker images -q -f dangling=true). Might take a while.
This keeps all tagged images around because they don't take up very much space anyway, and will become untagged during the next build so the cron job will clean them at midnight.
There is a Cron job on our Jenkins machines that runs these commands at midnight every day. This is probably too often, but disk space is much less of an issue if run regularly. The line looks like this:
0 1 * * * docker stop $(docker ps -q) || docker rm $(docker ps -a -q) || docker rmi $(docker images -q -f dangling=true)
Shove it in your root crontab with sudo crontab -e. If you've set Docker up to work with non-root permissions, you should edit the relevant user's crontab instead.
Thoughts
Our Docker images are quite small (they're based on Alpine Linux which is a fantastic base image, by the way) so disk space savings are modest, however if you're working with the Ubuntu base image, the space savings can make a real impact. This is not the most elegant solution to cleaning up Docker images, but it works well enough for our needs.
A possibly better way to go about this problem is to clean up old images straight after a build has run. No Cron jobs are required, but it removes the ability to debug old images which is useful sometimes.
Here's a disk usage comparison. Don't ask why we have a 1TB disk on a build server. AWS is cheap.
Initial disk usage:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 1008G 34G 933G 4% /
After cleaning up 50 or so images:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 1008G 24G 944G 3% /