My own Heroku in 30 mins

rails-google-compute-deploy.md

Deploy Rails apps to Google Cloud Compute Engine

• Zero Downtime
• Graceful shutdowns
• Via Github Actions
• Zero infrastructure management overhead

Overview

The general idea is to have Github Actions test, build and deploy a Rails app to Google Cloud Compute Engine. It works like this:

1. Push rails app to master branch.
2. Github Actions test, build and push a Rails app Dockerfile to the Google Container Registry.
3. Github Actions run database migrations.
4. Github Actions instruct Google Instance Group Manager to do a rolling update to the latest Dockerfile.

It's a poor man's solution to avoid and having to manage Kubernetes or using Google App Engine with computational restrictions. At the end of the day, this approach just deploys a Dockerfile to a Google Compute Instance running Google's Container Optimized OS.

Components

• Github Actions, including the deploy action and Dockerfile Rails Buildpacks
• Google Secret Manager, including google_cloud_env_secrets gem to load secrets at runtime
• Google Container Registry
• Google Instance Groups to manage deploys
• Google Compute Engine virtual machines
• Google Container Optimized OS
• Google Instance Templates
• Google Load Balancer
• Google CloudSQL Postgres Database
• Google Memorystore for Redis (optional)

Getting started

1) Create Google Secrets

Create Google Secrets which will be available to the Rails app as ENV variables. Choose a prefix for every secret, like MY_APP. For example: MY_APP_RAILS_DATABASE would save the connection string for the Postgres database in the Google Secrets Manager, but would still be available as RAILS_DATABASE to the Rails app.

• Set your prefix for GOOGLE_SECRETS_PREFIX in the rails.yml Github Action (see below)
• Set your prefix for SECRETS_PREFIX in the deploy.yml file (see below)

To connect the CloudSQL proxy to your CLoudSQL instance later, please also set <prefix>_CLOUDSQL_INSTANCE, i.e. MY_APP_CLOUDSQL_INSTANCE=my-project:region:my-instance.

2) Prepare Rails App

Add gem 'google_cloud_env_secrets' to your Gemfile and run bundle install. This Rubygem will load Google Secrets as ENV vars at runtime.

Since we are using Google Secrets, config/credentials.yml.enc won't be used anymore. Delete config/credentials.yml.enc and in config/environments/production.rb set config.require_master_key = false. You won't need to provide a RAILS_MASTER_KEY going forward, but you will have to create a Google Secret for MY_APP_SECRET_KEY_BASE which was previously provided by the credentials file.

Should secrets be stored as ENV vars? HackerNews discussed it. We are following the 12factor approach here. Using Google Secrets Manager also helps with tooling. For example: If you're using Terraform to provision your CloudSQL Postgres database, you can use it to easily write the connection string to a Google Secret MY_APP_RAILS_DATABASE. This would be exponentially harder if the secrets would be stored inside your Git repository.

3) Create Google Service Account

Create a Google Service Account for deploys, it will need the following roles:

• Compute Admin
• Service Account User
• Cloud SQL Client
• Storage Admin
• Secret Manager Secret Accessor
• Secret Manager Viewer

Create a JSON key for the service account you just created and copy and paste the contents to a Github Secret called GOOGLE_APPLICATION_CREDENTIALS. No need to base64 encode the secret.

4) Set up Github Actions

Copy the following file to .github/workflows/rails.yml. It will:

1. Create a local Postgres database (port 5432) and redis instance (port 6379) for testing.
2. Download Dockerfile Rails Buildpacks
3. Build Dockerfile.Base which is a base image with Ruby and Node installed
4. Build Dockerfile.Build which is based on Dockerfile.Base and also has Rubygems and NPM packages installed
5. Run rspec
6. If it's not the master branch, skip the next steps.
7. Precompile Rails assets
8. Build Dockerfile.Deploy which is a distroless image that contains the final Rails app.
9. Push the image to the Google Cloud Container Registry
10. Run database migrations
11. Start rolling update of the Google instance group

name: Rails 

on: [push]

jobs:
  pipeline:
    name: Build, Test & Deploy  
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:11
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: rails_test
          POSTGRES_HOST_AUTH_METHOD: trust
        ports:
        - 5432:5432
        options: --health-cmd pg_isready --health-interval 3s --health-timeout 5s --health-retries 5

      redis:
        image: redis
        ports:
        - 6379:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 3s
          --health-timeout 5s
          --health-retries 5

    steps:
    - uses: actions/checkout@v2

    - name: Fetch Dockerfiles
      run: wget -q -P /tmp https://raw.githubusercontent.com/mattes/dockerfile-buildpacks/v1.0.0/rails/Dockerfile.{Base,Build,Deploy}

    - name: Build Dockerfile.Base
      uses: mattes/cached-docker-build-action@v1
      with:
        args: "--file /tmp/Dockerfile.Base --tag rails:base ."

    - name: Build Dockerfile.Build
      uses: mattes/cached-docker-build-action@v1
      with:
        args: "--file /tmp/Dockerfile.Build --tag rails:build ."
        cache_key: ${{hashFiles('*.lock')}}

    - name: rspec
      run: >
          docker run --rm --net host -v $PWD:/app
          -e RAILS_ENV='test' -e NODE_ENV='test'
          rails:build
          'bundle exec rails db:create && bundle exec rspec'

    - name: Deploy - rails assets:precompile
      if: github.ref == 'refs/heads/master'
      run: >
          docker run --rm --net host -v $PWD:/app
          -e GOOGLE_APPLICATION_CREDENTIALS='${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
          -e GOOGLE_SECRETS_PREFIX=MY_APP
          rails:build
          'bundle exec rails assets:precompile'

    - name: Deploy - Build Dockerfile.Deploy
      if: github.ref == 'refs/heads/master'
      run: docker build -f /tmp/Dockerfile.Deploy --build-arg VERSION="${{github.run_number}}-${{github.sha}}" -t rails:deploy .

    - name: Deploy - Push Docker Image
      if: github.ref == 'refs/heads/master'
      uses: mattes/gce-docker-push-action@v1
      with:
        creds: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
        src: rails:deploy
        dst: gcr.io/my-project/my-app:${{github.run_number}}-${{github.sha}}

    - name: Deploy - Google Cloud SQL Proxy
      if: github.ref == 'refs/heads/master'
      uses: mattes/gce-cloudsql-proxy-action@v1
      with:
        creds: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
        instance: my-project:us-central1:my-instance
        port: 5433

    - name: Deploy - rails db:migrate
      if: github.ref == 'refs/heads/master'
      run: >
          docker run --rm --net host -v $PWD:/app
          -e GOOGLE_APPLICATION_CREDENTIALS='${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}'
          -e GOOGLE_SECRETS_PREFIX=MY_APP
          -e RAILS_DATABASE_PORT=5433
          rails:build
          'bundle exec rails db:create; bundle exec rails db:migrate'

    - name: Deploy - Start rolling update
      uses: mattes/gce-deploy-action@v5
      if: github.ref == 'refs/heads/master'
      with:
        creds: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}

This Github Action workflow uses a couple of helper files and actions. You can read more about them here:

• Dockerfile Buildpacks for Rails
• mattes/cached-docker-build-action
• mattes/gce-docker-push-action
• mattes/gce-cloudsql-proxy-action
• mattes/gce-deploy-action

5) Create deploy.yml

Please copy the following file to the root directory of your Git repository ./my-repo/deploy.yml where it will be picked up by the gce-deploy-action step and deploys app and worker.

The deploy action will:

1. Clone an existing instance template (using it as a base).
2. Update metadata config of the newly created instance template to run a cloud-init instructions file.
3. Tell the instance group manager to perform a rolling update with the new instance template.

Please review the configuration options and get a better understanding how this step works in detail. Please also review the rails helper utility used in deploys.*.vars.RUN which starts a Rails Docker Image.

common:
  region: us-central1
  cloud_init: https://raw.githubusercontent.com/mattes/gce-boot-scripts/v1.0.1/rails/dist/cloud-init.yml
  vars:
    DOCKER_IMAGE: gcr.io/my-project/my-app:${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
    SECRETS_PREFIX: MY_APP
  labels:
    version: ${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA}}
    git-sha: ${{GITHUB_SHA}}

deploys:
  - name: app
    instance_group: my-instance-group
    instance_template_base: my-base-instance-template
    instance_template: my-app-${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA:0:7}}
    vars:
      RUN: >
        rails --service my-app --run "rails server -b 0.0.0.0 -p 3000" --port 3000 -e RAILS_SERVE_STATIC_FILES=true;

  - name: worker
    instance_group: my-worker-instance-group
    instance_template_base: my-base-worker-instance-template
    instance_template: my-app-worker-${{GITHUB_RUN_NUMBER}}-${{GITHUB_SHA:0:7}}
    update_policy:
      type: OPPORTUNISTIC
    vars:
      RUN: >
        rails --service worker --container worker-1 --run "rails resque:work QUEUE='*'" --healthcheck-port 7000 --graceful-shutdown 300 --stop-signal SIGQUIT;
        rails --service worker --container worker-2 --run "rails resque:work QUEUE='*'" --healthcheck-port 7001 --graceful-shutdown 300 --stop-signal SIGQUIT;

Default Service Account Roles

Please make sure that the default service account used by the VM has role Secret Manager Secret Accessor. It must be added explicitly. This is for the default service account, not the service account you created for the Github Actions.

Graceful shutdowns

You'll notice how deploys.worker.update_policy.type is set to OPPORTUNISTIC. For app it defaults to PROACTIVE. What does that mean?

First it's important to understand that app is a request-based application serving short-lived HTTP requests, whereas worker runs long-running background jobs. Heroku would call app a Dyno, and worker just worker. How does it affect graceful shutdowns?

App

App and its Instance Group my-instance-group needs to use a Google Load Balancer with connection draining enabled. This ensures that existing, in-progress requests are given time to complete when a VM is removed from an instance group. During a deploy new instances are started, while the old instances finish their pending requests and then shut down. This enables zero downtime deployments. Easy.

Worker

For workers it's a bit more complex, because unfortunately Google Shutdown scripts are not reliable and at best only can run for a couple of seconds. Using a graceful-shutdown helper and setting the update policy to OPPORTUNISTIC we can use --graceful-shutdown [sec] to manually handle the shutdown procedure ourselves.

In the example shown above, --graceful-shutdown 300 is set to 5 minutes. When a new deploy happens the Docker container will receive SIGQUIT (set by --stop-signal) and then has 5 minutes guaranteed to stop. If the container doesn't stop after this, it will receive a SIGTERM signal but is essentially allowed to finish to run until the instance is finally deleted by Google. Please note that the "delete this instance" call is made by the graceful-shutdown helper, not the Google Instance Group Manager itself, hence the OPPORTUNISTIC deploy type.

To trap a signal inside Ruby, you could use:

should_exit = false
Signal.trap('TERM') { should_exit = true }

while true; do
  break if should_exit
  # process
end

Healthcheck helper

If what you're running doesn't provide a health check out of the box, you can use --healthcheck-port [port] to start a helper utility which returns HTTP status code 200 if the docker container is running. If it's not running it will return HTTP status code 500.

6) Create Google components

Following the example deploy.yml from above, you will need to create:

app

• Google Cloud Load Balancer with Connection Draining enabled
• Instance Template my-base-instance-template
• Instance Group my-instance-group (with or without autoscaler)
• Auto healing healtcheck for port 3000

worker

• Instance Template my-base-worker-instance-template
• Instance Group my-worker-instance-group (with or without autoscaler)
• Auto healing healtcheck for port 7000
• Auto healing healtcheck for port 7001

Troubleshooting

• I need a rails console
  SSH into any VM and just run console.

• Log files?
  SSH into a VM and tail -f /var/log/cloud-init-output.log or check the Google Logs UI.