DataXu is excited to share its solution for automating builds, testing and deployments. Our Automation Infrastructure uses GitHub, Jenkins, Ansible and AWS.
We had our Jenkins sitting in a closet and was manually configured with a few slaves. The jobs would poll for any changes and run unit tests if it caught anything. If someone was working on a new branch, they would create a new job to test that branch. This is a system that worked for us mostly, but as we grew, we ran into a number of challenges.
A commit was not tested and was merged to master. This throws a team off and developers may base new work off a bad state.
Jenkins jobs are a hassle to manage. We had a lot of jobs that were similar, so manually updating all of them when we needed to was cumbersome and error-prone. It was the wild west - anyone could change a job, we had no idea who did, what they changed or why they did it.
We were always running out of Jenkins capacity, resulting in some builds waiting hours to run. Developers need quicker feedback.
Jenkins would go down, slaves would be stale and builds would fail, resulting in sometimes days wasted. Slaves might need to be rebuilt or some hard-drive space cleared. It is always something.
Machine provisioning and deployments were being done manually, with people sshing into machines and running commands off a wiki page. This is time consuming and error prone.
We planned the next generation of our automation infrastructure and came up with this list of requirements.
- Automatically test everything before it gets to master
- Automated Job configurations must be stored in git
- Jenkins must be able to scale automatically. We want to have as much capacity as we need, but not keep extra machines running all the time. Save $$$$$, don't waste time!
- The system must be robust. Issues should not persist. If there is a weird issue, re-run or kill machines, don't waste my time making me look into it.
- All machines must be provisioned and deployed to using some configuration management. This includes Jenkins.
We identified these tools:
- Github-Webhooks - Triggers Jenkins Jobs based on Github Events. Created at DataXu.
- Github-PR - Allows programatic use of GitHub pull requests
- Jenkins Job Builder - Configures all Jenkins Jobs. Templates and macros allow reuse between jobs.
- Jenkins Amazon EC2 Plugin - Spins up and down jenkins slaves based on demand.
- Jenkins Build Flow Plugin (deprecated, see Workflow Plugin) - Allows orchestration of multiple jobs together.
- Ansible - Configures AMIs, including Jenkin Master and Jenkins Slaves. Runs deployments.
Note: This includes code examples. They are just rough examples, to be used for reference. Read the Ansible and Jenkins Job Builder documentation for more information. Our Jenkins is used as an example here, but this all can be generalized for your individual applications.
The Open Source community is awesome and there are roles that can bootstrap your setup. See ansible-role-jenkins.
This is just a brief reference on AMI creation using Ansible, not a comprehensive guide:
- name: create instance
hosts: localhost
vars:
ec2_ami: ami-123456
key: jenkins
tags:
- ec2_create
tasks:
- name: create instance
local_action:
module: ec2
image: "{{ ec2_ami }}"
key_name: "{{ key_name }}"
wait: yes
register: ec2
- name: add host
local_action: add_host
hostname: "{{ item.id }}"
ansible_ssh_host: "{{ item[ip_key] }}"
ansible_ssh_user: ec2-user
groupname: jenkins
with_items: ec2.instances
- name: wait for instance
local_action:
module: wait_for
host: "{{ item[ip_key] }}"
port: "22"
delay: "60"
timeout: "320"
state: "started"
with_items: ec2.instances
- name: configure jenkins master
hosts: jenkins
remote_user: ec2-user
tags:
- configure
roles:
- role: ansible-role-jenkins
jenkins_plugins:
- git
- ssh
- ec2
tasks:
- shell: echo "Install other libraries needed for your system and push config files"
- pip: name="github-pr"
- name: save ec2 ami
hosts: localhost
tags:
- ec2_save
tasks:
- local_action:
module: ec2_ami
instance_id: "{{ groups['jenkins'][0] }}"
wait: yes
name: "Jenkins AMI"
description: "Jenkins AMI"
register: ami
- name: teardown ec2 instance
hosts: localhost
tags:
- ec2_teardown
tasks:
- local_action:
module: ec2
state: 'absent'
instance_ids: "{{ item }}"
with_items: groups['jenkins']
This is just a brief reference on AutoScaling Group Deployment using Ansible, not a comprehensive guide:
- name: Deploy Jenkins to AutoScaling Group
hosts: localhost
vars:
ec2_ami: "{{ ami_id }}"
tasks:
- name: create config file
template: src="./templates/config.xml.j2" dest="{{ playbook_dir }}/tmp/config.xml"
- name: upload config file
s3: mode=put bucket="dataxu-jenkins" object="config.xml" src="tmp/config.xml"
- name: configure launch config
ec2_lc:
name: 'jenkins_{{ ami_id }}'
image_id: '{{ ami_id }}'
key_name: "jenkins"
instance_type: "c3.xlarge"
user_data: "aws s3 cp s3://dataxu-jenkins/config.xml ~/config.xml"
- name: configure autoscaling group and rolling-replace instances
ec2_asg:
name: 'jenkins-asg'
desired_capacity: '1'
min_size: '1'
max_size: '1'
launch_config_name: 'jenkins_{{ ami_id }}'
replace_all_instances: yes
replace_batch_size: 1
Now we have an instance of Jenkins running in AWS.
If you want elastic builds, be sure to configure the ec2 module during the deployment or manually on the running instance after. The benefits of this plugin are:
- Builds should not wait around too long, instead of just sitting in line, a new slave will spin up and take the job.
- If you usually have a lot of slaves, you can save some money by making them ephemeral.
- Slaves should stay in a cleaner state. If one is in a bad state, delete it. A fresh one should always spin up from the AMI you generated.
We also connect to our internal LDAP for authenticating and permissioning users to run jobs.
Now, Jenkins Job Builder goes to work. Create a git repository named something like, jenkins-jobs. Keep your job configurations in here in /jobs/ and a config at the top level, jenkins.ini
Install JJB locally and create your first job, to test and reconfigure Jenkins Jobs:
- job:
name: jenkins-jobs_master
node: master
scm:
- git:
url: 'git@github.com:dataxu/jenkins-jobs.git'
branches:
- 'master'
builders:
- shell: |
virtualenv my_env
. ./my_env/bin/activate
pip install jenkins-job-builder
jenkins-jobs -l debug test jobs
jenkins-jobs --conf jenkins.ini update jobs
Test this locally and deploy it with the above commands. Now, if you look to jenkins, there should be a jenkins-jobs_master you can run manually.
We use Github-Webhooks to automaically trigger Jenkins based on events in GitHub. Run github-webhooks on an aws instance that is internet-accessible. You should limit access to this instance to be just from github and your internal infrastructure. See github's documentation on whitelisting.
Once the instance is online, you need to test github's connection to it and start sending events. Go to https://github.com/organizations//settings/hooks and configure a hook to send to it. Additional documentation here.
We configure Github-Webhooks to try to trigger a repo's master job on every master commit.
push:
- match:
owner: "dataxu"
branch: "master"
trigger:
job: "{repo}_master"
Now, make a commit to your jenkins-jobs repository, add a new job and commit to master. See that your jenkins-jobs_master job is triggered and that jenkins is updated.
Great, now you have automated Jenkins Configuration! Let's trigger jobs to run against Pull Requests to your jenkins-jobs repo. Create the following job:
- job:
name: jenkins-jobs_pr
node: master
parameters:
- string:
name: 'fork'
- string:
name: 'branch'
scm:
- git:
url: 'git@github.com:${fork}/jenkins-jobs.git'
branches:
- '${branch}'
builders:
- shell: |
virtualenv my_env
. ./my_env/bin/activate
pip install jenkins-job-builder
jenkins-jobs -l debug test jobs
And add the following to your github-webhooks configuration:
pr:
- match:
actions:
- opened
- synchronize
trigger:
job: "{repo}_pr"
params:
fork: "{owner}"
branch: "{branch}"
Now you have a Jenkins infrastructure that allows you to easily create and review job configurations and test any incoming changes before they reach master. If you installed and configured the ec2 plugin, you will also have jenkins slaves that scale up and down as you need them.
Read up on how to use jenkins-job-builder you can set up your jenkins jobs to:
- Email developers on failure
- Report the status of the commit back to github using Github Status API
- Use a common template for building all java, python, ruby projects - This can really save time during the development of new projects.
Refer back to the Jenkins AMI and Deployment sections. They are simple ansible playbook that were run locally. Let's make jobs that create and deploy AMIs. First, put your ansible playbook in a git repo as jenkins_create_ami.yml and jenkins_deploy_ami.yml. Let's call the repo, ansible-playbooks. Now we can create generic jobs that use those playbooks using job-templates.
- job-template:
name: "{app}_create_ami"
parameters:
- string:
name: 'fork'
default: dataxu
- string:
name: 'branch'
default: master
scm:
- git:
url: 'git@github.com:${{fork}}/ansible-playbooks.git'
branches:
- '${{branch}}'
builders:
- shell: |
virtualenv my_env
. ./my_env/bin/activate
pip install ansible
ansible-playbook {app}_create_ami.yml
- job-template:
name: "{app}_deploy_ami"
parameters:
- string:
name: 'ami_id'
- string:
name: 'fork'
default: dataxu
- string:
name: 'branch'
default: master
scm:
- git:
url: 'git@github.com:${{fork}}/ansible-playbooks.git'
branches:
- '${{branch}}'
builders:
- shell: |
virtualenv my_env
. ./my_env/bin/activate
pip install ansible
ansible-playbook {app}_deploy_ami.yml -e ami_id=${{ami_id}}
- job-group:
name: ami-jobs
jobs:
- "{app}_create_ami"
- "{app}_deploy_ami"
- project:
name: my-app
app: jenkins
jobs:
- 'ami-jobs'
Now we can run jenkins_create_ami and jenkins_deploy_ami jobs in jenkins to create a new jenkins ami and deploy it.
Jenkins Build Flow Plugin has been deprecated in favor of the Workflow Plugin. This example is how we use it now. New implementations using the Workflow Plugin may be similar.
This job will run a job we have for build/unit test, create an AMI, test that AMI and then deploy that AMI whenever there is a commit to the myapp master branch.
- job:
name: 'myapp_master'
project-type: flow
needs-workspace: 'true'
scm:
- git:
url: 'git@github.com:dataxu/myapp.git'
branches:
- 'master'
dsl: |
build('myapp_build', branch: 'master')
ami_job = build('myapp_create_ami')
artifact = ami_job.getArtifacts()[0].getHref()
url = ami_job.getEnvironment()['BUILD_URL']
ami_name = new URL("${url}artifact/${artifact}/*view*/").getText().tokenize()
ami_id = ami_name[1]
build('myapp_test_ami', ami_id: ami_id)
build('myapp_deploy_ami', ami_id: ami_id)
Note: We are just just reaching this phase and are using this for a limited number of repositories.
If you have enough confidence in your tests, a pull request has passed all necessary testing could be merged immediately. We created a tool to look up pull requests in GitHub, github-pr. We report back to Github using the Status API, so we can look up the status using github-pr.
$ github-pr list -r dataxu/myapp --table
# State Status Merge Base Head Title
--- ------- -------- ------- ------------ ------------- -----------------------
794 open success clean master some-branch added something awesome
$ github-pr merge -r dataxu/myapp -n 794
We can use this script to do a number of things in an automated in a job that runs nightly.
- Look up all successful branches, merge them into an integration branch for testing before merging to master
- Merge Pull Requests
- Add labels to pull requests
- Comment the status back to pull requests
Everything runs through Jenkins. We automatically trigger builds on commits to master and to pull requests. By running Ansible, we can configure jobs to automatically create AMIs and deploy those AMIs. Job flows can be used to hook jobs together. Job configurations are stored in SCM, so we are aware and can approve of all changes before merging them.
By setting standard jobs that exist, we don't need to reinvent the wheel every time there is a new project, we just need to fill in the gaps.