Setup an external private image registry with hasura kubernetes cluster

setup-external-registry-hasura.md

Setup an external private image registry, with Hasura k8s platform

If you are trying to install Hasura on a multi-node Kubernetes cluster and wondering how to setup the image registry, this is the guide for you.

Let us see how to setup a private image registry on a multi-node Hasura k8s platform cluster. This is required in multi-node setups, because the sshd agent (which builds the docker images on git push) needs to push the image to an external image registry service, so that the image is available on all the nodes.

NOTE: Do these steps before installing Hasura on the Kuberentes cluster.

Setup a private registry

1. Setup an account in your private registry.

2. Login to that account using docker CLI (Make sure you have latest enough docker (>= 17)).

$ docker login

This will generate a config.json file in ~/.docker directory. That contains an auth token, which we can use as our credentials in the Kubernetes cluster. Ideally do it on a fresh system account/different computer, so as to avoid adding all of your accounts into the Kuberentes cluster.

3. Verify:

$ cat ~/.docker/config.json

It should contain something like:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "c3R...zE2"
        }
    }
}

Add the credentials to the Kubernetes cluster

0. Assumption: you already have a multi-node Kubernetes cluster; and kubectl setup on your local machine and configured to access your cluster.

1. Create the Kubernetes secret on the cluster (in the hasura namespace)

kubectl create secret generic registry-creds \
    -n hasura \
    --from-file=.dockerconfigjson="~/.docker/config.json" \
    --type=kubernetes.io/dockerconfigjson

Alternatively, if you want more control, you can create the secret from a YAML file.

apiVersion: v1
kind: Secret
metadata:
  name: registry-creds
  namespace: hasura
data:
  .dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
type: kubernetes.io/dockerconfigjson

The value of .dockerconfigjson has to be base64 encoded string of the ~/.docker/config.json file.

You can get that by running:

$ cat ~/.docker/config.json | base64 -w 0

After creating the file, you can create the secret by running:

$ kubectl create -f <path-to-secret-file.yaml>

Setup the registry for Hasura

1. Edit your cluster-data.yaml of your project.

2. Add the registry key, under metadata key:

infra:
  provider: custom
...
metadata:
  namespaces:
    hasura: hasura
    user: default
  registry:
    dockercfgSecret: 'registry-creds'
    prefix: 'docker.io/<username>'
    # or in case of GKE
    prefix: 'gcr.io/<gcp-project-name>'
  gateway:
    ports:
...

Now you can use this cluster-data.yaml to install Hasura on your K8s cluster.