Everyone uses Git today. I can hardly image how one would write a piece of software without it. And I can hardly remember any alternatives to Git. This is why I strongly feel that all parts of a project must be version controlled. Not only the application, but all the infrastructure. And especially the deployment pipelines. All the scripts on your Jenkins. All the little knobs and valves on your AWS account. And all the manual stuff people do on a daily basis that should have been automated two years ago.
And there are tools that do that. There is Terraform to get your infrastructure up and running. If your Jenkins is not hilariously outdated, it supports pipelines as code. You can write your code, build and deploy it with Jenkins. Or even deploy it with git hooks.
But then there is this need for mutliple deployment stages. From your customers point of view this is very understandable. They want a prelive environment where the new features can be accepted by a product owner. And this should better not be the live environment. And you might also benefit from such an environment. You can test things out in the wild. Perhaps even on a dedicated development environment.
You might be pushing for a one stage solution, to deploy everything directly to the live system. And I am with you. But in the meantime many of us are stuck with multiple deployment environments.
So you need a mechanism that knows when, how and why to which environment software can be deployed. And while the project evolves, these rules might change. They might change from incident to incident. Yes, this complicates things!
The tool we ended up using came with a lot of features. We knew that if we wanted to do canary releases we would be covered. As complex as you could imagine a deployment pipeline to be, the tool could handle it. It also featured a nice web client that allowed you to really drill down into your deployments. You could fiddle with all the settings directly in the browser. This was nice. The power and the feeling of having everything under control.
Until someone accidentally deleted the deployment settings of a critical service. And we noticed that there has not been a deployment in days. Or instead of typing in slack someone mistakenly typed into the environment variable configuration of the product information importer. And it won't start anymore. Rolling back is not that easy and that one guy that understands the correct settings just left the office. These are the times when you want a version controlled solution. One that allows for easy roll backs, and disallows accidents.
We investigated other solutions. Some seemed too hacky, others overengineered. But none truly had a version controlled approach. So I wrote my own.
The name spacegun is a pun on the deployment manager of a project I used to work on. And because space guns are pretty awesome and ridiculous. Just look at this.
Spacegun can be installed as a server, but it also comes with tools for controlling all deployments from the command line. Installation is done via npm install spacegun and the setup process should be mostly straightforward once you understand the basic principles.
Visit spacegun tutorial for a more hands-on tutorial
All configuration of spacegun is in yaml files and should reside in a git repository. A running spacegun server instance will then periodically look into the configuration repository and checkout new commits from the master branch. So any configuration change will lead to spacegun picking up on it automatically. The basic configuration does not change that often: the docker endpoint, the available cluster, the slack hook...
But the configuration repository can hold so much more:
You can add your deployment pipelines in a pipelines/ folder. A typical deployment pipeline might look like this:
cluster: k8s.live.my-app.com
cron: "0 0 0 12 * * MON-FRI"
start: "plan"
steps:
- name: "plan"
type: "planClusterDeployment"
cluster: "k8s.staging.my-app.com"
onSuccess: "apply"
- name: "apply"
type: "applyDeployment"
This yaml file describes a deployment to the live cluster k8s.live.my-app.com. The cron statement tells spacegun to execute the pipline at 12 am from monday through friday and that it should start with with the plan step. The plan step creates a list of updates by looking at the staging cluster k8s.staging.my-app.com. If it succeeds it proceeds to apply the plan.
Let's assume you have some kind of incident. Feelings are hurt and policies changed. Now deployments to the live environment need to be approved by the product owner and always done manually. To do that you would just remove the cron expression, make a pull request and once the change is merged spacegun will pick it up automatically. Then any time everyone is happy with the current staging you would deploy it with spacegun run.
Running spacegun snapshot creates a snapshot of the current state of your cluster. Spacegun holds this snapshot as kubernetes yaml files in its configuration repository. If one of your teammates wants to add a new environment variable, she would first change the appropriate configuration files and make a pull request. And just like changes to a deployment pipeline, spacegun will automatically apply them.
Even if you add a new deployment. You would just add a new yaml file that describes the deployment and after having it reviewed by your colleges, spacegun will create the deployment in kubernetes. If you have the slack hook configured spacegun will even inform you about it.
The goal is to automate the manual things we have to do each day. A good step forward is adding more steps to the pipeline. A step that probes the cluster for its health, a step that rolls back a cluster to a previously stable state, a step that seeks approval via slack... There are still many things to do.
We are also experimenting with templating of the configuration and I am always working on stabilizing and extending the current functionality. We just started to use it in a production environment on our current project.