Setup an google cloud image registry with hasura kubernetes cluster on GKE

setup-gke-registry-hasura.md

Setup Google cloud image registry with Hasura k8s platform on GKE

This guide helps you to setup the image registry configuration on a multi-node Hasura installation on GKE.

This is required in multi-node setups, because the sshd agent (which builds the docker images on git push) needs to push the image to an external image registry service, so that the image is available on all the nodes.

Pre-requisite

1. gcloud CLI (https://cloud.google.com/sdk/install)
2. kubectl (https://kubernetes.io/docs/tasks/tools/install-kubectl/)

gcloud setup

First make sure are logged in: gcloud login

List all the projects:

gcloud projects list

And then set the correct project on which you have the Hasura cluster:

gcloud config set project <project-id>

Create a service account on Google Project and with appropriate permissions

Run the below steps:

# create a GCP service account; format of account is email address
SA_EMAIL=$(gcloud iam service-accounts --format='value(email)' create hasura-gcr-auth --display-name hasura-gcr-auth)
# create the json key file and associate it with the service account -> DON'T CHECK-IN TO VERSION CONTROL!
gcloud iam service-accounts keys create hasura-gcr-auth.json --iam-account=$SA_EMAIL
# get the project id
PROJECT=$(gcloud config list core/project --format='value(core.project)')
# add the IAM policy binding for the defined project and service account
gcloud projects add-iam-policy-binding $PROJECT --member serviceAccount:$SA_EMAIL --role roles/storage.admin

Create a kubernetes secret from the service account credentials

Make sure your kubectl is pointing to the right cluster. (Use kubectl config use-context <cluster-context> to set it to the correct cluster).

1. Put this in a script, in the same directory as above, and run:

#!/bin/bash
set -e
_auth="_json_key:$(cat hasura-gcr-auth.json)"
auth=$(echo $_auth | base64 -w 0)
cat <<EOF > _tmp_dockercfg.json
  {
    "https://gcr.io": {
      "username": "_json_key",
      "email": "a@example.com",
      "password": "$(cat hasura-gcr-auth.json | sed 's/"/\\"/g' | sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g')",
      "auth": "$auth"
    }
  }
EOF

Then create a Kubernetes secret from the service account credentials file and above output.

kubectl create secret generic registry-creds \
  --from-file=.dockercfg="_tmp_dockercfg.json" \
  --type kubernetes.io/dockercfg

Setup the registry for Hasura

1. Edit controller-conf:

kubectl -n hasura edit configmap controller-conf

2. Add the following lines inside the cluster.json key under data. (Use the project-id from previous steps)

{
  "prefix": "gcr.io/<project-id>",
  "dockercfgSecret": "registry-creds"
}

Make sure the entry under cluster.json key is a valid JSON string.

So the controller-conf should look something like this:

apiVersion: v1                                                                                                                          
data:                                                                                                                                   
  cluster.json: |                                                                                                                       
    {                                                                                                                                   
      "namespaces": {"user":"default","hasura":"hasura"},                                                                               
      "gateway": {"selector":{"app":"gateway"},"ports":[{"targetPort":80,"protocol":"TCP","name":"http","port":80},{"targetPort":443,"pr
otocol":"TCP","name":"https","port":443},{"targetPort":22,"protocol":"TCP","name":"ssh","port":22}],"type":"LoadBalancer","loadBalancerI
P":"35.274.50.140"},                                                                                                                    
      "postgres": {                                                                                                                     
        "volume": {"name":"postgres","gcePersistentDisk":{"pdName":"postgres"}}                                                         
      },                                                                                                                                
      "sessionStore": {                                                                                                                 
        "volume": {"name":"sessionstore","gcePersistentDisk":{"pdName":"sessionstore"}}                                                 
      },                                                                                                                                
      "filestore": {                                                                                                                    
        "volume":  {"name":"filestore","gcePersistentDisk":{"pdName":"filestore"}}
      },
      "volumes": {"postgres":{"name":"postgres","gcePersistentDisk":{"pdName":"postgres"}},"sessionstore":{"name":"sessionstore","gcePersistentDisk":{"pdName":"sessionstore"}},"filestore":{"name":"filestore","gcePersistentDisk":{"pdName":"filestore"}}},
      "registry": {"prefix":"gcr.io/painkilling69","dockercfgSecret":"registry-creds"}
    }
  controller-conf.json: |
    {
      "controllerNamespace": "hasura",
      ...

If everything is well, you should be able to now push your microservices to Hasura and they will get built and pushed to your custom registry.

git push hasura master