Deployment on AWS Elastic Beanstalk

blog.md

Context

AWS Elastic Beanstalk is the Platform As A Service solution by AWS. It is thought as a replacement for Heroku wich is great 🙌 but can become very expensive 💰 when the project is growing or in need of real performance 🏎.

Google App Engine comparison

Google Cloud also has its Platform As A Service product, Google App Engine. However very promising and notably simpler than AWS Elastic Beanstalk, we have been experiencing serious issues while testing it:

• downtime in deployment 🚧: when deployong a new version of the app, it often becomes unreachable (5xx errors) for up to a whole minute.
• veeeery long deploys ⬆️ ⏲: it takes on average 8 to 10 minutes ⏳ waiting for a new version of the application to deploy. Compared to a few seconds on Heroku and a few minutes on Elastic Beanstalk. We found out that it truly harmed our development workflow, leading to more context switching, less deploys and tests.

Prerequisites

Interacting with Elastic Beanstalk can be done either of 2 ways:

• using AWS Console for Elastic Beanstalk 🕹
• using Elastic Beanstalk CLI ⌨️, eb cli.

  For some interactions (e.g. deploy your app) it is mandatory to use the CLI

Install EB CLI

Install eb cli 🛠 on Mac OS with Homebrew 🍺:

https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-install-ebcli-bash

Initialize an EB CLI project

After installing eb cli you need to configure it 👷‍♂️ for your project and with your credentials 🔑.

https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-eb-init-bash

Set up

Application

1. Create a New Application 🧙‍♂️on the AWS Console for Elastic Beanstalk. You'll need to set an application name (e.g. goldengate)

   IMPORTANT ⚠️: as with everything you do on AWS, carefully choose the region 🌍 you start your application on. It's not possible to change afterwards. Even when exploring the AWS console, make sure you are actually looking in the right region, or you won't see your application 🤷‍♂️

2. Create an environment 🏜 for your application. Environments are essentially instances of your application. Environments can have different configurations (number and size of EC2 instances, databases, environment variables, etc.)

   environments should be created using eb cli ⌨️

https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-eb-create-bash

Database

To add a database to the newly created application, AWS provides integration with AWS RDS (Relational Database Service). AWS RDS is AWS's DB As A Service and offers MySQL, PostgreSQL, Oracle, or SQL Server databases.

Create a database

Adding a database to your application comes in 2 options:

1. create the database in your environment 🏠

   the database is "linked" to the environment; i.e. if the environment is destroyed, the DB is destroyed as well

2. launch a database outside of your environment 🏕 and link it to your environment.

Option 1. - database in our environment - seems like a reasonable choice since data is hardly shared between environments (e.g. production and staging).

Create a Database in the environment

Create the database In AWS Elastic Beanstalk console 🕹 using the documentation An RDS instance created this way will automatically expose the following environment variables to your application: RDS_HOSTNAME, RDS_PORT, RDS_DB_NAME, RDS_USERNAME, RDS_PASSWORD

These environment variables should be used in your rails application: https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-rails-database-yaml

Your application should now be able to connect to the database ☕️

Create an external database

Create the database in AWS RDS console using the documentation. Choose the Default VPC approach (aka nothing to do).

The whole process can be summed up in:

1. create the database 🧙‍♂️ in RDS
2. modify security rules 👮‍♂️(i.e. security groups) of the newly created DB to allow Elastic Beanstalk to connect
3. manually create the RDS environment variables 🏜 in Elastic Beanstalk holding credentials to connect to your database (e.g. RDS_DB_NAME, RDS_DB_NAME, etc.)

Database gotchas

Some information is either hidden or a sensible default and can be hard to find. The database name 📛 is always ebdb.

Sidekiq

Setting up Sidekiq on Elastic Beanstalk requires using .ebextensions feature. .ebextensions is a way to create advanced environment customization with configuration files.

Here is a sidekiq.config .ebextension file which seems to be working (not yet tested)

Migrate from Heroku

When migrating from Heroku, you need to migrate 🚛 data from Heroku Postgres to the Postgres DB on AWS RDS.

Dump your Heroku DB

Follow Heroku Documentation on exporting ⬇️ a Postgres database: https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-heroku-backup-bash

Import the dump to AWS

Temporarily open public access

This requires temporarily making the RDS database Publicly Accessible 🔓. In Inbound rules of your RDS instance grand access to Anywhere 0.0.0.0/0 on TCP 5432 port

⚠️ this will need to be reverted afterwards because it's far too much permissive

Check connection

You need to make sure you can connect 🔗 to AWS RDS using native commands (i.e. psql and pg_restore )/ Check if you can psql into your account; you will be prompted for your password: https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-psql-rds-bash

• RDS_USERNAME: the username you entered when creating the RDS DB instance
• RDS_HOSTNAME: the full hostname of the database such as aape7pwctyqeox.cfynf9xjh4ys.eu-west-3.rds.amazonaws.com

Note: the dbname is always ebdb; yes that's a gotcha 😉🤷‍♂️

Import the dump

Import the dump ⬆️ to your RDS instance using pg_restore: https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-rds-pg_restore-bash

Source: How to migrate your postgresql database from heroku to Amazon RDS

Usage

Deploy

Deploy ⬆️ a new version of your application using eb deploy. 📡

IMPORTANT ⚠️: eb deploy deploys the current state of your local git repository (branch and HEAD). It essentially means it won't deploy any non commited change (this can be quite confusing at first 😅)

rails console

To run rails console 👨‍💻 on an Elastic Beanstalk application:

1. ssh into your instance using eb ssh
2. run the following to run rails console (source)

https://gist.github.com/nicolasrouanne/4bd107e8f7d46011518d1487961f4d1b#file-rails-console-bash

eb-create.bash

$ eb create
# Following will be a set of questions about the environment (name, platform version, etc.)

eb-init.bash

$ eb init
# following will be a set of questions asking credentials, region, platform etc.

heroku-backup.bash

# make a new backup
$ heroku pg:backups:capture
# download the last backup
$ heroku pg:backups:download

install-ebcli.bash

$ brew update
$ brew install awsebcli
# check that eb cli is correctly installed
$ eb --version

psql-rds.bash

$ psql -U RDS_USERNAME -h RDS_HOSTNAME --dbname=ebdb

rails-console.bash

$ cd /var/app/current
$ sudo su
$ rake rails:update:bin
$ bundle exec rails console

rails-database.yaml

# config/database.yml
production:
  adapter: postgresql
  encoding: unicode
  database: <%= ENV['RDS_DB_NAME'] %>
  username: <%= ENV['RDS_DB_NAME'] %>
  password: <%= ENV['RDS_PASSWORD'] %>
  host: <%= ENV['RDS_HOSTNAME'] %>
  port: <%= ENV['RDS_PORT'] %>

rds-pg_restore.bash

$ pg_restore --verbose --clean --no-acl --no-owner -h RDS_HOSTNAME -U RDS_USERNAME -d ebdb ~/path/to/database/dump