Terraform + Ansible notes

gistfile1.txt

# Terraform + Ansible

## My assumptions about you:

- You've used neither of these tools.
- You may have used AWS before.
- You get the whole infrastructure in the cloud thing.

It's already available for you to play with right now — please grab it and ask questions!

---

## What Cognician needed:

- Approachable infrastructure code for the whole tech team.
- A safe testing environment that matches production exactly, but shares nothing with it — true 'infrastructure as code'.
- The ability to make more environments without much ceremony.
- A fresh start that left old assumptions and systems behind.
- Therefore, to stop using the Cthulhu that is CloudFormation+Bash.

---

## What I chose:

- Cobus' excellent advice to keep separate environments in separate AWS accounts, with a controlling 'master' account for common elements, like domain registration, Terraform's state and team IAM accounts.
- Use Packer to build a single general-purpose base AMI. All custom servers get the same basics.
- Use Terraform for provisioning infrastructure. 
- Use Ansible playbooks for instance configuration.

---

## Why Packer and the single AMI?

- All the 'fiddly bits' are baked into the AMI, in a single linear bash script.
- Installed packages - Java, Python, Ansible, Datadog's agent, CodeDeploy's agent, etc.
- Conveniences - bash aliases, hostname changes,  etc.
- All of these things change very rarely.
- So that none of it has to happen during instance startup.

---

## Why Terraform?

- You declare what you want, and it compares that to what's live, and makes the necessary changes to make your wishes the new 'normal'. This makes it the tool's responsibility to get that right, rather than us; which means that we don't reinvent these wheels (again).
- It's new enough that becoming involved in its community is easy and leads to great general guidance.
- Although it can be buggy, it releases very frequently and matches today's AWS very closely.
- Easy to roll out improvements — e.g. switch from S3-based to CloudFront-based Route53 failover took a couple hours to do for all web apps.

---

## Terraform - how?

- Generic modules that model the vertical that an app needs - from DNS record to instances.
  - VPC. Bastion instance. App. Web app. Etc.
- As needed, each environment invokes modules with its own inputs.
- As needed, each environment directly declares extra bits like S3 buckets and additional DNS records.
- Each environment tracks its own variables, and merges them with base variables common to the whole business (such as the AWS region, master account ID, app names, etc).

---

## Ansible

- Why?
  - Script the sequence of things that take an instance from the base AMI state to the state needed for the app.
  - These scripts can change often, and so code clarity and maintainability are more important.
- Run during auto-scaling, and can also be run from dev workstation against existing instances.
- Common and app-specific environment variables.
- App-specific process management.
- Starting up Datadog, Papertrail, CodeDeploy agents once everything's in place.

---

## How this all composes

- Terraform outputs values that it captures during its update.
- A script copies some of these into the Ansible playbooks.
- Another script puts the playbooks on S3 for autoscaling to use.
- Each environment has a makefile holds it all together.

---

## It's extracted!

It's a mix of the things Cognician needs. In no particular order, those are:

- A single VPC, with the usual 3 security group setup - internal, web, and bastion. AWS accounts separate the env VPCs from each other.
- A bastion a.k.a. jump host.
- A Datomic transactor pair.
- Elasticache:
  - Memcached - Datomic's 2nd-tier peer cache.
  - Redis - a function memoisation backend using https://github.com/strongh/crache.
- A Zookeeper cluster.
- Several S3 buckets.
- Several web apps.
- Several non-web apps.
- AWS CodeDeploy.
- Datadog and Papertrail integration.