spinnaker-on-gke.md

Spinnaker on GKE

This serves as a reference of how I configured a distrubuted installation of Spinnaker running on GKE. It assumes a certain environment although you may pick and choose from this guide.

Specifics:

• A seperate GCP project for Cloudbuild (useful if you orchestrate across different project)
• Use GCS for storage
• Use CloudBuild and PubSub to trigger pipelines
• Use GCR as container registry
• Use identity aware proxy for Google authentication
• Use Helm chart to deploy application called website

Setup

• Give Cloudbuild project Cloudbuild SA storage object creator permissions Needed because builds will create artifacts in a bucket

• Create GKE cluster for Spinnaker On hardware requirements: https://www.spinnaker.io/setup/install/environment/

• Create SA for Spinnaker Spinnaker uses account (which contain credentials) to authenticate with different services

  • For storage: storage admin in same project
  • For artifact account: storage admin permissions on Cloudbuild project
  • For PubSub account: pubsub client permissions
  • For k8s pull secret: cloudbuild project GCR image pull permissions
  • For GCR account: project browser + storage admin permissions

• Initialize Helm (tiller) for the created cluster

• Provision Halyard somewhere where it has access to the Spinnaker cluster (https://www.spinnaker.io/setup/install/halyard/)

• Enable K8s provider (https://www.spinnaker.io/setup/install/providers/kubernetes-v2/)

  • Create k8s serviceaccount that Spinnaker will use to interact with its own cluster (https://www.spinnaker.io/setup/install/providers/kubernetes-v2/#optional-create-a-kubernetes-service-account)
  • Similarly, create k8s serviceaccount for clusters it needs to deploy to and add them as accounts

• Configure GCS storage (using created SA) (https://www.spinnaker.io/setup/install/storage/gcs/#editing-your-storage-settings)

• Configure GCS artifact account (using created SA) (https://www.spinnaker.io/setup/artifacts/gcs/)

• Create PubSub subscription for Cloud Builds topic (https://www.spinnaker.io/setup/ci/gcb/#configure-spinnaker-to-listen-for-google-cloud-build-pubsub-notifications)

• Configure PubSub triggering (using created SA) (https://www.spinnaker.io/setup/triggers/google/#editing-your-pubsub-settings)

• Configure GCR as container registry (https://www.spinnaker.io/setup/install/providers/docker-registry/#google-container-registry)

• Create bucket in Cloudbuild project for Helm chart artifacts from builds

• Deploy (https://www.spinnaker.io/setup/install/providers/kubernetes-v2/#adding-an-account)

• Expose UI (https://www.spinnaker.io/setup/quickstart/halyard-gke-public/)

  • Over HTTPS
  • With OAuth2 authentication

• Optional but recommended: manually skip over all existing cloudbuild images by port forwarding spin-igor port 8080 to localhost and running:

  curl -X POST localhost:8080/admin/pollers/fastforward/dockerTagMonitor?partition=${gcr-account}

Configure

• Create pipeline with cloud build trigger using created PubSub subscription (https://www.spinnaker.io/setup/ci/gcb/#configure-your-pipeline-trigger)

  • Specify status SUCCESS as payload constraint

  • Specify buildTriggerId as payload constraint

  • Add tag parameter Allows for manually triggering the pipeline

    • Required
    • Name should be tag
    • Set default value to: ${trigger['payload']['source']['repoSource']['tagName']}

  • Add expected artifacts:

    • Helm chart

      • Specify the object path: gs://[ARTIFACTS_BUCKET]/website/charts/v[0-9.]+.tar.gz
      • Specify the default: gs://[ARTIFACTS_BUCKET]/website/charts/${parameters.tag}.tar.gz

    • Container(s)

      • Specify Docker image:

        gcr.io/[GCR_REPO]/website

      • Specify the default: gcr.io/[GCR_REPO]/website:${parameters.tag}

• Add bake stage to pipeline

  • Select HELM2 as template renderer
  • Specify name (Helm release name)
  • Add GCS artifact from previous stage as expected artifact
  • Enable “fail stage on failed expressions”
  • Add base64 as produced artifact and give it a name

• Add deploy stage to pipeline

  • Select account that corresponds to the desired cluster
  • Select base64 artifact from previous stage as expected artifact
  • Add containers as required artifacts to bind This will override the container versions in the Helm chart

Modify app

• Add Helm chart in infra/helm/{CHART_NAME}
• Add cloudbuild.yaml with
  • Build Docker image
  • Optionally tag image with version
  • Archive Helm chart to ${TAG_NAME}.tar.gz and upload to the cloudbuild project charts bucket under the correct app folder
  • Add created archive and images as artifact

Example cloudbuild file

steps:
    # Build image
  - name: 'gcr.io/cloud-builders/docker'
  	args: ['build', '-t', 'gcr.io/$PROJECT_ID/website', '.']
  	
    # Tag with version
  - name: 'gcr.io/cloud-builders/docker'
  	args: ['tag', 'gcr.io/$PROJECT_ID/website', 'gcr.io/$PROJECT_ID/website:$TAG_NAME']

    # Create chart archive
  - name: 'gcr.io/cloud-builders/gsutil'
    entrypoint: bash
    args:
      - '-c'
      - |
        tar -czvf ${TAG_NAME}.tar.gz -C infra/helm/website .

# Specify chart archive artifact
artifacts:
  objects:
    location: 'gs://[ARTIFACTS_BUCKET]/website/charts'
    paths: ['$TAG_NAME.tar.gz']

# Specify image artifacts (at least one without tag for Spinnaker to pick it up)
images:
  - gcr.io/$PROJECT_ID/website
  - gcr.io/$PROJECT_ID/website:$TAG_NAME