OpenShift integrated registry with redis as a memcache

README.md

OpenShift integrated registry with redis as a memcache

motivation: find out if it works with multiple replicas with image pruning.

worry: are the entries safely removed from cache when image pruning is run?

Tested with:

oc v3.7.14
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Setup

1. deploy cluster with registry (advanced install is assumed)
2. login as cluster admin
3. oc project default
4. oc new-app --name=registry-redis --docker-image=docker.io/redis
5. oc rsh dc/docker-registry cat /etc/registry/config.yml >config.yml
6. sed -i 's/blobdescriptor: inmemory/blobdescriptor: redis/' config.yml

cat >>config.yml <<EOF
redis:
  addr: $(oc get -o jsonpath='{.spec.clusterIP}:{.spec.ports[0].port}' svc/registry-redis -n default)
EOF

8. oc secret new registry-redis-config config.yml

oc volume dc/docker-registry --add \
    --type=secret --secret-name=registry-redis-config \
    --name=docker-config --overwrite

10. oc scale --replicas=2 dc/docker-registry
11. oc rollout status

Trying it out

• Monitor for db operations in one terminal

  oc exec -it "$(oc get pods -o jsonpath='{.items[0].metadata.name}' -l deploymentconfig=registry-redis)" \
      -n default redis-cli monitor
  

• Push image:

  oc new-project redistest
  docker login -u unused -p $(oc whoami -t) $REGISTRY_SERVICE_IP
  docker pull busybox && docker push $REGISTRY_SERVICE_IP/redistest/busybox
  

• List the keys:

  oc exec -it "$(oc get pods -o jsonpath='{.items[0].metadata.name}' -l deploymentconfig=registry-redis)" \
      -n default redis-cli keys \*
  1) "blobs::sha256:8072a54ebb3bc136150e2f2860f00a7bf45f13eeb917cca2430fcd0054c8e51b"
  3) "blobs::sha256:f2a91732366c0332ccd7afd2a5c4ff2b9af81f549370f7a19acd460f87686bc7"
  4) "repository::test/hello-world::blobs"
  5) "repository::test/hello-world::blobs::sha256:ca4f61b1923c10e9eb81228bd46bee1dfba02b9c7dac1844527a734752688ede"
  6) "blobs::sha256:ca4f61b1923c10e9eb81228bd46bee1dfba02b9c7dac1844527a734752688ede"
  7) "repository::test/hello-world::blobs::sha256:f2a91732366c0332ccd7afd2a5c4ff2b9af81f549370f7a19acd460f87686bc7"
  

• Prune the pushed image:

   oc adm --token=$(oc whoami -t) prune images \
       --force-insecure=true --keep-tag-revisions=0 \
       --keep-younger-than=0 --all=false --confirm
  

• The prune does not delete any key. It attempts to delete blobs::$digest but due to a bug, the keys stays, nevertheless the attempt is enough for the key to be considered as invalid.