Last Thursday CircleCI had a security breach, and now many companies are struggling, because not only do they have to rotate their credentials for one SaaS provider, but now they have to rotate ALL THE THINGS in their production and non-prod environments because EVERY SECRET EVERYWHERE for EVERY ENVIRONMENT just got leaked. Because they were all in CircleCI. Because you used CircleCI to deploy at the end of your CI step. Deploy to test. Deploy to staging. Deploy to production. And now all your base are belong to 1337 h4xx0r5.
This has caused a lot of people in a lot of companies a lot of pain.
Here are N simple strategies to enable you to avoid this pain:
- Don't use SaaS CI/CD
- Use SaaS CI but not to deploy
So, strategy 1. is run your own pipelines, on your own hardware. Please don't use Jenkins, it really is horrible. I've set up internal CI/CD using Drone, it was much simpler and lower overhead than Jenkins, I'm sure there are many alternatives that are equally good. Hell, a Makefile would be better than Jenkins.
For many people though, that is more work than they have person-hours to invest, the whole point of SaaS after all is to speed things up.
So option 2: use SaaS CI, but don't trust it with the keys to your kingdom(s).
flowchart LR
A[I build and test]--things-->B[I deploy];
At Gower.st I built a simple tool that we called the Deployerer. It was a containerised service that responded to a webhook that was invoked when a new docker image was uploaded to our private DockerHub repo. It then deployed that image to the correct environment, complete with all the secrets and config that were necessary for it to operate. It did this through existing container management functionality, there was no wheel reinvention. How exactly I made it work isn't relevant, because there are many different service orchestration platforms - although in practice most of you will be using K8s. Sorry.
This is not anything new, clever or revolutionary. It is basic Separation of Concerns. Your CI server handles build, test, and uploading of deployment artifacts to a store of some kind. You can outsource this to a SaaS provider if you want. A CD server takes deployment artefacts and deploys them to your environments, for which it needs access. This is all it does. Do not outsource this. Ever.
Follow this simple advice and you will avoid losing the next month of your life to rotating security credentials for all your apps and services in all your environments.
Agree but also note that this does depend on your threat model. If you are using a SaaS to build your docker images, a compromise of that SaaS might mean that an attacker could backdoor those images and then when you later deploy them into production you're now running bad code.
This is obviously a bit more convoluted for the attacker than just collecting private ssh keys for prod. It could be made simpler if e.g. the CI service were also a cache/proxy for your third-party dependencies: then the attacker can backdoor the cached copy of httparty or left-pad or some other library and have that code baked into (almost) every customer's golden images not just yours. Some build systems are more tamper-resistant than others, also: yarn.lock contains a SHA sum for each download, but Gemfile.lock contains nothing of the sort.