flapjack packaging ideas

packaging.markdown

The SQS and Jenkins bits can come later...

Create an AMI

• ubuntu trusty with docker PPA and latest docker installed
• do a 'docker pull flapjack/omnibus-ubuntu'
• bake ami

Start EC2 instance using above AMI

• user data with #!/bin/bash on first line to run the flapjack build commands ie docker run ...
• the docker run commands will include calls to the scripts that update packages.flapjack.io S3 hosted website with the newly built package

SQS queue and an ASG

• create an autoscaling group that runs 1 ec2 instance of the docker build server and takes the details of the package to produce as parameters
• an ASG policy based on an alarm that fires when the queue has >= 1 message
• so the ASG will process jobs when there are jobs on the queue to be processed
• for bonus points, do this with CloudFormation

Jenkins or some such

• create jobs on the SQS queue as required based on git commits

So, to sketch out and test the usefulness of SQS based auto-scaling prior to cloudforming the whole thing:

• In the SQS Console, create an SQS queue with parameters that sound appropriate. There are lots of behavioural subtleties here, but go with whatever to start with because we're only mucking around. :-)
• In the CloudWatch console, click "Create Alarm"
• Select a metric:
  • You should see an "SQS Metrics" section. If so, click. If not, wait for up to five minutes.
  • Select the "ApproximateNumberOfMessagesVisible" metric -- this is essentially the size of the queue.
  • Click "Next".
• Define the alarm:
  • Give that sucker a name and description. I'd suggest making the name something like [project]-queue-has-items or something.
  • The default is >= 1 for 1 period, so that's good.
  • Delete the default action (top right of the box under Actions).
  • Choose a period of 5 minutes (that's how often SQS reports) at the bottom right.
  • The only valid statistic for that metric is Average, so leave it as is.
  • Click "Create Alarm".
• Now we need to make an auto-scaling policy that uses your new alarm!
• In the EC2 Console, navigate to Auto Scaling Groups.
• Choose your builder ASG.
• Switch to the Scaling Policies tab
  • Click "Add policy".
  • Give it a nice name, such as [project]-scale-out.
  • Execute policy when: Choose the alarm you created here.
  • Take the action: Set to 1 instance.
  • And then wait: Let's chat about this one... it gets interesting. For now, leave it at 60s, because we don't have any "scale in" policies yet.
  • Click "Create".

Done! :-)

Awesome, that'll keep me busy for a while.