Clean Docker Registry

gistfile1.md

Summary

The following directory /var/lib/registry/docker/registry/v2/blobs/sha256 may be filling up with abandoned, unused docker blobs setting off an alert for /var exceeding the monitoring threshold for filesystems.

In the Gitlab web interface, on the Registry tab, when you click on the "delete" button, only the image tag is actually deleted, and the data itself does not go anywhere and continues to take up space on the hard disk. A similar problem is observed if the docker image is created with the same tag (for example, latest) — and this is common for the CI process. In this case, when building a new sample, the old one is not deleted (as expected), it simply loses the tag (latest). For example, the image actually has 2 versions stored on the hard disk:

ls /data/registry/docker/registry/v2/repositories/project/repo-name/_manifests/tags/
1.73.67  2.66.2  2.67.1  2.70.5  2.71.8   2.72.12  2.72.9  2.73.4  2.73.8   2.74.11  2.74.14  2.74.17  latest
2.65.3   2.66.3  2.70.3  2.71.6  2.72.10  2.72.13  2.73.2  2.73.5  2.74.1   2.74.12  2.74.15  2.74.18
2.66.1   2.66.4  2.70.4  2.71.7  2.72.11  2.72.14  2.73.3  2.73.6  2.74.10  2.74.13  2.74.16  2.74.19
...
ls /data/registry/docker/registry/v2/repositories/project/repo-name/_manifests/tags/latest/index/sha256/
0fab781bf8f185d59cb58a3b974736f756f31e72ae3b9c7402f8f2cd2f55611e/
6f4d92727f4f18586f4f9542de6db04a2f7582bbe567324b879d85532c0e0eb5/
...
ls /data/registry/docker/registry/v2/repositories/project/repo-name/_manifests/revisions/sha256
0fab781bf8f185d59cb58a3b974736f756f31e72ae3b9c7402f8f2cd2f55611e/
6f4d92727f4f18586f4f9542de6db04a2f7582bbe567324b879d85532c0e0eb5/

To clear the space in the private docker-registry, follow these steps::

• delete old versions of docker image tags;

• delete old versions of docker image revisions;

• start the process of "garbage collection" in the container with docker-registry

To perform the cleanup, run this script: ./clean_docker_registry.sh /path/registry/repositories

# Add # docker run -d ... -e REGISTRY_STORAGE_DELETE_ENABLED=true --name registry ..., or # docker exec -it registry sh, and # env |grep REGISTRY_STORAGE_DELETE_ENABLED=true, if not then add # REGISTRY_STORAGE_DELETE_ENABLED=true
# Use: # ./script.sh /PATH/repositories

#!/bin/sh

###
[ -z $1 ] && bash -c "echo Not correct REPOPATH=$1; echo Use: $0 /REPOPATH" && exit 1
[ ! -d $1 ] && bash -c "echo Not correct REPOPATH=$1; echo Use: $0 /REPOPATH" && exit 1

Clean () {
    for R in $(ls -t ${1}/_manifests/tags/ | tail -n +2); do
        TAGPATH=$1/_manifests/tags/$R/index/sha256
        REVPATH=$1/_manifests/revisions/sha256
        for hash in $(ls $TAGPATH -t | tail -n +2)
        do
            rm -rf $TAGPATH/$hash;
            rm -rf $REVPATH/$hash;
            echo -e "rm: $TAGPATH/$hash\n$REVPATH/$hash"
        done
    done
}

for D in $1/*; do
    if [ -d "${D}" ]; then
        if [ ! -d ${D}/_manifests/tags/ ]; then
            for E in ${D}/*; do
                if [ -d ${E}/_manifests/tags/ ]; then
                    Clean $E
                else
                    for F in ${E}/*; do
                        if [ -d ${F}/_manifests/tags/ ]; then
                            Clean $F
                        else
                            for G in ${F}/*; do
                                if [ -d ${G}/_manifests/tags/ ]; then
                                    Clean $G
                                else
                                    echo -e "Not found ${G}/_manifests/tags/: => Skipped"
                                fi
                            done
                        fi
                    done
                fi
            done
        else
            if [ -z "$(ls -A ${D}/_manifests/tags/)" ]; then
                echo ''
            else
                Clean $D
            fi
        fi
    fi
done
# Version
# docker exec $(docker ps | grep registry | awk '{print $1}') registry -v
# Parameters
# docker exec $(docker ps | grep registry | awk '{print $1}') /bin/registry garbage-collect -h
# NOTE: The --dry-run option garbage-collect prints the clean up plan without removing any data
# Works in the version v2.6.1
# [ $? = 0 ] && docker exec $(docker ps | grep registry | awk '{print $1}') /bin/registry garbage-collect /etc/docker/registry/config.yml
# Works in the version v2.7.1
# [ $? = 0 ] && docker exec $(docker ps | grep registry | awk '{print $1}') /bin/registry garbage-collect /etc/docker/registry/config.yml --delete-untagged=true
[ $? = 0 ] && docker exec $(docker ps | grep registry | awk '{print $1}') /bin/registry garbage-collect /etc/docker/registry/config.yml -m
echo Done!

In the process, you will see approximately: ...

project/repo-name: marking blob sha256:bc10e760fcc0085df23aead50df0996d279fc82585613c488f65055bd53pv8r1 ...

time="2021-01-27T17:52:05Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/3a/6f4d92727f4f18586f4f9542de6db04a2f7582bbe567324b879d85532c4h6ds8" go.version=go1.7.3 instance.id=6e41a878-0fb3-47a6-8eb7-ce3e1a71b567

Done!

To clear the registry in K8s, something like: https://www.ibm.com/support/pages/how-clean-abandoned-unused-docker-blobs-ibm-cloud-private