read_me.md

environment_provisioning

###Summary This project will let you build an new environment from scratch in AWS.

It uses the python library Troposphere to generate Cloudformation templates, which are uploaded to S3. The url of the template on S3 is then passed to the Cloudformation service, which builds the environment.

###Deploying a new copy of the default environment

The core of this is "build_environment.py". A successful run does not output anything. Log into AWS and look at the Cloudformation console to watch your environment being built.

• -n --name : The name of your new environment. This will become part of the URLs for your services. "-n myname" would create "www.myname.mntd.net" for the web servers. This is also the name that Cloudformation will show for your stacks.

• -i --input : Name of a file to load parameter values from. Must be JSON formatted, and JSON keys must match input parameters in the templates. Values in this file override the default values in the templates.

• -I --iam : REQUIRED If you do anything with IAM in your stack. This appends the IAM flag that passes CAPABILITY_IAM parameter to CloudFormation, and is required to make any changes to IAM resources. This defaults to false.

• -r --region : The AWS region you want to deploy your environment to. Defaults to 'us-east-1'.

• -t --token : Your MFA token number for AWS login. If you don't have MFA enabled on your account, this script won't work.

• -c --credentials : Your AWS credentials file. Defaults to ~/.boto and follows the same formatting rules.

• -D --dump : Skips the upload & creation steps, instead just dumps the generated templates to stdout. Useful for tracking down problems in your templates.

• -g --gitsha : Sha of the current commit. Tags stack with version of environment_provsioning repo. If no sha is supplied, this param defaults to git rev-parse HEAD.

At the end of the run, you will have:

• 1 new VPC
• 2 new Subnets in the VPC (Private & Public)
• 1 SSH Gateway host in the Public Subnet
• Several app servers, all in the Private Subnet
  • 2 web servers
  • 2 mbo servers
  • 1 printer-api server
  • 1 minted-api server
  • 1 packslip server
  • 1 solr server
  • 1 mbocrons server
  • 1 webcrons server
  • 1 jenkins server
• Several load balancers to serve traffic from the above app servers:
  • web
  • mbo
  • printer-api
  • minted-api
  • solr
  • jenkins
• All the various routing & security groups that are necessary for this system to be secured.

Puppet should have run at server startup, so once they are showing as "up", they will be ready to be deployed.

The names of the load balancers & the IP address of the SSH Gateway instance are part of the outputs of the template, and will appear in the "Outputs" tab on the Cloudformation console.

###Creating/updating the templates

Make sure you've got a firm understanding of Troposphere and AWS Cloudformation before you start, or your pain shall be heard throughout time.

You can build almost anything in AWS with this. There are lots of easy ways to trip yourself up, so here are some things I found helped me keep the chaos to a minimum:

• AWS limits what you can do. For example, if you try to upload a template body directly to the Cloudformation service, it can only be 51,200 bytes in size. See the full list of limits here.
• Do use the same name for a variable no matter what template you're working in.
• Do use mappings for parameters that have different values based on other parameters (but see the limits above). Not everything needs a mapping, but for things like AMIs it means you won't have to look it up.
• Do try to keep your template as simple as possible. Simpler templates are more reusable, readable, and easier to repair.
• Do use the stack system where appropriate. Stacks of stacks can help you isolate issues quickly, while still keeping everything in a cohesive architecture. There is a max size on templates too.
• Do put tags on everything you can, and almost everything can be tagged.
• Do add an output for anything you might want to know about later. (Also see limits above)
• Do expect that it won't build correctly the first time. Or the 10th.

Notes on Message_service

Use Virtual Environments

• It is encouraged that for development, every project should have a virtualenv to avoid dependency version conflicts. For more information and documentation of virtualenv, please refer to here.

• The python version on most of the macbooks in Minted is 2.7.9. This will lead to "ssl.CertificateError" when you try to create a message_service stack. To resolve that issue, follow the listed steps:

  cd environment_provisioning

  virtualenv -p /usr/bin/python2.7 venv

  run python --version to ensure that you are not using python 2.7.9

• After this setup, every time when you want to work on this project, you should source this environment by running:

  source venv/bin/activate

  The folder venv should be the one under THIS PROJECT, not anyone else.

• If it is the first time you work on this project, with virtualenv activated, install all the dependencies by running:

  pip install -r pip-requirements

• If you want to leave this project and work on another one, you can leave this environment by running:

  deactivate

Set up AWS credential

• Refer to the wiki page about how to setup awsauth.

Create your template

• Create your own template in folder <PROJECT_PATH>/message_service/dev-.json

Create your own stack

• Using Supplied Makefile:

  • You can use the supplied Makefile run the two steps listed below into a dev-stack:

    make ENVIRON="dev-<username>" GIT_TAG="<Tag-That-AMIs-Were-Built-With"
    

• Create your own stack by running:

  cd <PROJECT_PATH>

  vim message_service/dev-<yourname>.json # To Change AMI ID's To the AMI's You've Pre-Created

  python ./build_environment.py -s message_service -e messaging_service -i message_service/dev-<yourname>.json -n MessageService-dev-<yourname> -I -w

TODOs:

• Add unit tests to the code.
• Add some kind of template testing prior to kicking off Cloudformation.
• Add some utility commands, such as:
  • --delete (delete a stack)
  • --status (check stack status)
  • --plan (get an idea of exactly what's going to be created before you create it.)
• Add parameter validation to default environment.
• Add more documentation to code.
• Maybe a better guide on how to use this thing :)