So far we have made great strides at moving towards creating more micro services vs monolithic applications. With Docker + Elastic beanstalk we are pushing the boundries and setting a trend for future compaines to follow. As more people use these tools we will continue to discover new challenges. One challenge we began to see immediately was how to deploy these microservices.
Our deploy tool (shipit) does not currently support the abbility to deploy these applications. Cukebot (our integration testing server) runs in Elastic Beanstalk and is continuously deployed via AWS's EB cli tool. We curl the EB cli package to the travis-ci vm and set some configuration files on the fly in order to gain access to the "git aws.push" command. Also this requires all of our developers to do the same. The configuration files can not be checked into subversion so each dev would have to simmulate recreation of the app using "eb init" in order to deploy code changes.
Docker has released an awesome feature allowing you to upload and store your applications images. They also have an automated build feature that will listen to a specified github branch for commits. When an update is made on a branch the build is triggered in dockerhub and an image is created from the dockerfile in the repo. You can specify a tag for the image thats autobuilt.
Instead of using AWS's EB cli (which doesn't support deploying of a dockerrun file) I created a gem called gantree which will allow you to create a new Dockerrun.aws.json using "docker init" and deploy any image tag that was automatically built from docker hub to your beanstalk application (versioned and tagged according to github hash).
Where we will need to go from here is create another server (like geoffrey) that listens for images that are automatically built, checks the travis status for that github hash, then deploys the image using gantree to a configured staging environment, notify our chatroom of deploys then runs integration tests.
This solves our deployment issues however some new challenges that we can/should solve.
The issue we have now is if someones micro service stack is corrupted and needs to be updated progromatically it needs to be in a cloud formation template. Elastic Beanstalk uses Cloud Formation in the background but does not give you the same access of control as building the templates yourself. If I add some functionality to the gantree gem to generate clould fromation templates and allow developers an easy solution to select the resources they need in their solution stack.
I am not quite sure how to solve this yet but we have heard amazon could possibly add this as a deployment feature. If not I still would consider adding weighted deploy management inside gantree for blue green deploys and later create a solution to automated the weight increasing over time if we had propper monitoring for those stacks.
Currently the only thing that our new docker deployment flow does not do well is transparency. We need to add integrations for slack to show when a new build for each application is completed, and when it is deployed. Release notes need to be generated on each production deploy and added to our dev wiki (like shipit does). Shipit also surfaces the lighthouse tickets that are associated with each deploy to our cukebot service and deploy dashboard service so this needs to be replicated as well.
I feel our two strongest areas to attack as we move forward with these micro services is:
- Transitionint towards developing stacks via cloud formation nested templates generated through gantree.
- Build a new service to accept dockerhub api posts and notify slack of new deployments microservice deployments.