Skip to content

Instantly share code, notes, and snippets.

@JAMSUPREME
Last active March 24, 2020 16:22
Show Gist options
  • Save JAMSUPREME/638d4820ecf694a20ded866e756b88b8 to your computer and use it in GitHub Desktop.
Save JAMSUPREME/638d4820ecf694a20ded866e756b88b8 to your computer and use it in GitHub Desktop.
CI/CD Primer

CI/CD Primer

This is designed to be a quick primer on Continuous Integration and Continuous Delivery. We will also use the term "Continuous Deployment" to mean a subset of Continuous Delivery in which you are delivering with every commit.

Acronyms and Terms

II/ID: Intermittent Integration & Delivery

For the sake of this discussion, we should have a term for "The opposite of CI/CD" and we'll call that "Intermittent Integration" and "Intermittent Delivery" so that we have a shared vocabulary.

CI: Continuous Integration

You can read the textbook definition on wikipedia. For the intents of this discussion, we will assume it is the practice of integrating your code into a shared master, preferably with as many of the "best practices" as possible. I will outline an example below.

CD: Continuous Delivery

You can read the textbook definition on wikipedia.

For the intents of this discussion, let's broadly say it means You can release at any time (even if you do not).

CD: Continuous Deployment

For the intents of this discussion, let's broadly say it means You release with every commit

What's the point?

Faster feedback loops. You want everything to "shift left" (happen earlier). That means code review, basic QA testing, security, infrastructure, performance testing, etc. should all happen early and often in the Software Development Life Cycle (SDLC).

Agile vs. Waterfall

It is useful to think of CI vs. non-CI as a codified version of agile vs. waterfall. Let's imagine a "Golden Architecture" where everything is agile and CI/CD vs. a "Not-So-Golden Architecture" where everything is waterfall and II/ID.

Exposition

You have a basic web site that shows pictures of dogs (like instagram).

The Golden Architecture (agile)

  1. You get a requirement to show cat pictures in addition to dogs
  2. You implement it in a feature branch
  3. You make a PR to merge the code into master and it runs your CI suite (linting, tests, etc.)
  4. Everything passes and a teammate approves it
  5. The codes goes into master and gets deployed into prod
  6. Repeat this for every feature

The Not-So-Golden Architecture (waterfall)

  1. You get a requirement to show cat pictures in addition to dogs
  2. You implement it in a branch shared with 10 other developers
  3. You try to merge the code into master but you have a lot of merge conflicts from everyone else's code
  4. You think you solved the conflicts and merge it into master
  5. Your QA team tells you everything is broken
  6. A teammate tells you they had some unstable code in that branch, so you have to figure out how to revert their change
  7. You end up branching off an earlier release because the feature is urgent and you copy/paste your code there
  8. You realize your code is dependent on some recent changes not in the release, so you need to rewrite your code
  9. Repeat this for every feature

Go from Not-So-Golden to Golden!

Going from 0 to 100 takes a while, so there are a number of steps you can take (in no particular order) to help improve your iterative speed.

  • Use feature branches (git is preferable, but not mandatory)
  • Keep the scope of features small
  • Review code and run CI suite before it goes into the main branch
  • If you don't have a CI suite, make one
  • Ensure each developer's work environment is the same (e.g. use Docker Compose) to avoid framework mismatch blunders - this env should also be as similar to prod as possible

What should my Continuous Integration look like?

Let's go through an example of some "good things". Each of these could warrant their own entire 1 hour talks, but I'll touch on them so you have some general guidance.

  1. A CI server or setup of some sort. There are a number of frameworks (or combinations) you could use:
    • Jenkins
    • AWS CodePipeline
    • Atlassian Bamboo/Bitbucket & Octopus Deploy
  2. Unit tests in your application, preferably "Integration" tests for web apps:
  3. End-to-End tests
    • Webdriver
    • Cypress
  4. Configuration Management / Infrastructure-as-code. Ideally, you are also using some form of CM for your Jenkins, too!
    • Docker
    • Ansible / Chef / Puppet
    • Cloudformation
    • You should also test this when possible (e.g. chefspec, inspec, etc.)
  5. Database Management (and any other "stateful" pieces of your architecture)
    • Frameworks/libraries:
      • Ruby: Rails activerecord
      • Java: Liquibase / Flyway
    • You should also typically treat your cache (Redis/Memcached) much like a DB
    • Other long-lived items like an S3 bucket or EFS may also require careful management to ensure the correct convergence
  6. Performance tests and Application Performance Monitoring
    • Performance needs to be benchmarked, or you don't really know how well your app is performing. Tools like NewRelic, AppDynamics, Dynatrace, Raygun are useful for monitoring your apps.
    • Tools like JMeter (Blazemeter) / Loadrunner / Locust for running tests
    • Your site's performance is probably worse than you think (neat blog: https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/)
  7. Scaling / Availability

Continuous Deployment Pre-Reqs

There are a few must-have pre-reqs you should have in place before you go all-in on continuous deployment. Everything else is "nice to have" or can be built incrementally, but without these in place, you're probably going to fail.

  • Test suites - You need to have an automated test suite so you can trust your deploys. I recommend starting with a decent e2e suite since it will be the most "realistic" simulation of production.
  • Feature flags / "Toggles" - You need a way to push features into master without "accidentally" showing them to users before the code is ready.
  • Rollback / Roll forward - You need to have a strategy for quickly rolling back or rolling forward.

Other recommendations

Here are a few other good practices for moving towards safe, predictable, quick iterations into production:

  • Application Monitoring: AWS X-Ray, NewRelic, synthetics/canaries, etc.
  • Blue/Green (immutable) deployments: Deployments roll out into a new group to ensure no downtime (and can also go through an additional test suite)
  • Immutable infrastructure: You can make a docker image (or AMI) and this ensures you don't have drift on a particular instance, as well as enabling easy rollbacks if a newer deploy goes wrong.
  • Cattle vs. pets: Generally you want cattle (Clusters/Autoscaling groups, destroyed at any time) instead of a pet (long-lived Jenkins, cannot destroy without problems) so you can easily terminate any particular instance without worry. It also ensures you don't end up with drift.
  • Everything-as-code If it can be code, it should be code (or something code-like). This makes development and testing much easier. Examples:
    • Docker / docker-compose - This makes your image "code" and easy to replicate on any OS
    • Config. Management (chef) - This makes any modifications to the image (if not done via a Dockerfile) easy to replicate (and test)
    • Pipeline - No one wants to manually configure Jenkins and individual pipelines, especially after drift occurs
    • Infrastructure - With Cloudformation (or Terraform), creation and teardown are fairly simple and predictable. (OPINION: Prefer declarative over imperative, e.g. CloudFormation vs. Boto)
    • Database - Even if you want to have native SQL stored procedures, functions, etc., you should at least put them in version control and use some tool (ORM or otherwise) to correctly apply changes to them

Further reading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment