Skip to content

Instantly share code, notes, and snippets.

What would you like to do?

Container Image Registry

The Registry is managed by the Image Registry Operator, deployed to the openshift-image-registry project.

As of this writing (installer 0.12.0), the deployment follows the steps outlined in the manifest directory:

  1. Create the CRD
  2. Create the namespace
  3. Request credentials from the Cloud Credential Operator
  4. RBAC and ServiceAccount(s)
  5. Request certificate(s) from the Cert Signing Operator
  6. Create the cluster-image-registry-operator deployment
  7. Create the cluster operator object

Each of these depends on the previous being successful. Failure may not prevent future steps from creating the objects, but will result in the deployment not being successful for a myriad of reasons depending on the step which failed.

Registry Info

  • Viewing objects related to the registry instance All commands below are expected to be executed from within the project: oc project openshift-image-registry.

    • Secrets:
      # the secret for the custom storage S3 credentials:
      oc get secret image-registry-private-configuration-user -o yaml
    • Deployments:
      # the operator
      oc describe deployment cluster-image-registry-operator
      # the registry instance
      oc describe deployment image-registry
    • CRD:
      oc describe crd
    • Configuration:
      # this is where the operator configuration is applied
      oc get config instance -o yaml
    • ConfigMap(s):
      # only two config maps exist, they hold the certificates used
      oc get configmap
    • Route(s):
      oc describe route default-route
  • Checking the health of the registry deployment

    For the most up to date troubleshooting information, check the GitHub page for the operator.

    To verify if the registry has been deployed, check the status of the deployment:

    # check for the deployment
    oc get deployment image-registry -n openshift-image-registry
    # check the pods for the deployment
    oc get pods -n openshift-image-registry

    If you do not see the pod for the registry deployed, then you'll need to find out why deployment failed. Follow the instructions at the operator GitHub page linked above.

Modifying the registry configuration

The config/instance object in the openshift-image-registry controls the configuration which is pushed to the deployment. To change the configuration, edit the object (oc edit config instance -n openshift-image-registry) and set values in the spec section.

By default, with 0.12.0, the operator will configure these defaults:

  • One (1) replica
  • S3 storage in the same region which the cluster is deployed
  • The S3 bucket will be encrypted
  • Multipart uploads which fail will be cleaned after one (1) day

Some common values to edit/change:

  • Un-deploying the registry - Set the value of spec.managementState to Removed (set to Managed to resume default behavior)
  • Changing the number of replicas - Set the value of spec.replicas to your desired value, e.g. 3
  • Increase/decrease the log level - Set the value of spec.logging to your desired value: 0 = error, 1 = warn, 2 and 3 = info, all other values = debug
  • Managing storage configuration - Edit the values in Other storage types are not supported at this time. The following values can be modified:
    • bucket - the name of the bucket to use if the default generated name is undesired
    • region - the name of the region to use, e.g. us-east-1
    • regionEndpoint - the region endpoint, will use the default for the region specified above
    • encrypt - whether to encrypt the bucket or not, defaults to false, but the managed deployment (spec.managementState = Managed) will enable encryption for the bucket
  • Create a route so that the registry is publicly accessible - Set spec.defaultRoute to true
  • Specify a node selector for the pods - Set spec.nodeSelector to the desired value, e.g.


  • When using the 0.12.0 installer, it was discovered that the registry was not being successfully deployed. The reason for the failure was that the credentials object did not get created in the project by the Cloud Credential Operator (which had failed with an error about insufficient permissions).

    • Symptoms

      1. Verify that the image-registry pods (not just image-registry-operator!) are not deployed:

        oc get pod -n openshift-image-registry | grep image-registry | grep -v operator
      2. Check the status of the registry deployment

        oc get deployment image-registry -o yaml -n openshift-image-registry
      3. If the deployment doesn't exist, check errors at the operator

        oc get -o yaml

        At this point the errors revealed that the secret with the credentials did not exist.

    • Fixing the problem As documented on the operator page, providing custom credentials is supported by creating a separate secret:

      cat << EOL > registry-user.yaml
      apiVersion: v1
      kind: Secret
        name: image-registry-private-configuration-user
        namespace: openshift-image-registry
        REGISTRY_STORAGE_S3_ACCESSKEY: <access key>
        REGISTRY_STORAGE_S3_SECRETKEY: <secret key>
      type: Opaque
      # create the object
      oc create -f registry-user.yaml

      Or, create it from the CLI:

      export ACCESSKEY=< Your AWS Access Key>
      export SECRETKEY=<Your AWS Secret Key>
      oc create secret generic image-registry-private-configuration-user -n openshift-image-registry --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=${ACCESSKEY} --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=${SECRETKEY}
  • When using the 0.12.0 installer, after applying the above credential fix, the registry deployment failed due to no S3 region being supplied.

    • Symptoms:
      1. Check the status of the deployment:

        # the logs for the pods reported the error
        oc logs image-registry-XXX-XXX -n openshift-image-registry

        The above output reflected the error:

        time="2019-02-12T18:33:25.716442598Z" level=info msg="start registry" distribution_version=v2.6.0+unknown go.version=go1.9.4 openshift_version=v4.0.0-0.149.0
        time="2019-02-12T18:33:25.716923859Z" level=info msg="caching project quota objects with TTL 1m0s" go.version=go1.9.4
        panic: No region parameter provided
      2. Describing the pod showed that the parameter had not been defined:

        oc describe pod image-registry-XXX-XXX

        The describe looks like this, note the lack of a value for the environment variable REGISTRY_STORAGE_S3_REGION.

        Name:               image-registry-54db8bd7c4-9vpgt
        Namespace:          openshift-image-registry
        Priority:           0
        PriorityClassName:  <none>
        Node:               ip-10-0-31-5.ec2.internal/
        Start Time:         Tue, 12 Feb 2019 18:27:19 +0000
        Labels:             docker-registry=default
        Status:             Running
        Controlled By:      ReplicaSet/image-registry-54db8bd7c4
            Container ID:   cri-o://0bb77f94907ff7e6becf52c6e210ff547009bab893b356fc1f80931d4dfa4206
            Image ID:
            Port:           5000/TCP
            Host Port:      0/TCP
            State:          Waiting
              Reason:       CrashLoopBackOff
            Last State:     Terminated
              Reason:       Error
              Exit Code:    2
              Started:      Tue, 12 Feb 2019 18:33:24 +0000
              Finished:     Tue, 12 Feb 2019 18:33:25 +0000
            Ready:          False
            Restart Count:  6
              cpu:      100m
              memory:   256Mi
            Liveness:   http-get https://:5000/healthz delay=10s timeout=5s period=10s #success=1 #failure=3
            Readiness:  http-get https://:5000/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
              REGISTRY_STORAGE:                       s3
              REGISTRY_STORAGE_S3_BUCKET:             image-registry--91d67b3b845f4231a21e8cb912c13dbc-d45f0ab82ef31
              REGISTRY_STORAGE_S3_ENCRYPT:            false
              REGISTRY_STORAGE_S3_ACCESSKEY:          <set to the key 'REGISTRY_STORAGE_S3_ACCESSKEY' in secret 'image-registry-private-configuration'>  Optional: false
              REGISTRY_STORAGE_S3_SECRETKEY:          <set to the key 'REGISTRY_STORAGE_S3_SECRETKEY' in secret 'image-registry-private-configuration'>  Optional: false
              REGISTRY_HTTP_ADDR:                     :5000
              REGISTRY_HTTP_NET:                      tcp
              REGISTRY_HTTP_SECRET:                   <data>
              REGISTRY_LOG_LEVEL:                     info
              REGISTRY_OPENSHIFT_QUOTA_ENABLED:       true
              REGISTRY_STORAGE_DELETE_ENABLED:        true
              REGISTRY_OPENSHIFT_SERVER_ADDR:         image-registry.openshift-image-registry.svc:5000
              REGISTRY_HTTP_TLS_CERTIFICATE:          /etc/secrets/tls.crt
              REGISTRY_HTTP_TLS_KEY:                  /etc/secrets/tls.key
              /etc/pki/ca-trust/source/anchors from registry-certificates (rw)
              /etc/secrets from registry-tls (rw)
              /var/run/secrets/ from registry-token-dv64s (ro)
          Type              Status
          Initialized       True
          Ready             False
          ContainersReady   False
          PodScheduled      True
      3. Lastly, the config reflects the error:

        oc get -o yaml config instance -n openshift-image-registry

        Look for the current status in the status.conditions section of the output.

      4. The fix is to edit the deployment definition and set the value for to the region the cluster is deployed to, e.g. us-east-1:

        oc edit config instance -n openshift-image-registry

        Or, patch the config:

        oc patch config instance -n openshift-image-registry --type merge --patch '{"spec": { "storage": { "s3": { "region":"us-east-1"}}}}'
      5. After editing, OpenShift will redeploy with the updated configuration. Monitor the deployment using the same commands:

        oc get deployment
        oc describe deployment image-registry
        oc get pod | grep image-registry | grep -v operator
        oc describe pod image-registry-XXX-XXX
        oc logs image-registry-XXX-XXX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.