This model and approach is based on the belief that the (Docker) image of the application (and not the source code) should be the "source of truth" and object of the Continuous Integration / Continuous Deployment (CI/CD).
Key Points Covered...
-
Image-based deployment artifacts versus code-based
-
Images are built once and potentially promoted to production instead of being rebuilt when the feature branch is merged
-
This approach provides more certainty and efficiency than either code-based deployment artifacts or rebuilding the image on merge
-
Separates the concerns of Continuous Integration and Continuous Deployment
-
Adaptable to support multiple and/or multi-architecture images and additional security
This is basically the model...
Branch-based, commit-tagged images are idempotently built, vetted for code quality, and if passing, promoted to eventually become the production image upon merging. The image is never rebuilt, but its tag, name, and container repository change as it is promoted.
I use the term "vetting" to mean all of the automated "checks" like linting, security scanning, unit tests, and even acceptance tests and I use the term "promoted" to abstract this concept from the actual mechanics of how this is done. For example, for single-platform images, pulling, retagging, and pushing the image can be used to promote it. However for multi-platform images, a copy operation at the container-registry level might be used.
One of the biggest values of containerization technology is certainty and much of that certainty is lost when rebuilding the image on merge.
The big-picture value this model brings if you are already deploying images is in certainty and efficiency. Many CI processes rebuild the image upon merging and although the source code is the same, the image may not be the same as its dependencies like the base image and any installed libraries and packages may have changed, especially if these are not strictly versioned (e.g. using the image digest for the base image). This means that the image being deployed to production may not be the one that was tested. The other option is to run all the tests again on this newly-built image (but then what is the point of the testing done on the previous image). The model presented here eliminates this uncertainty and inefficiency of building a new image upon merge.
🙇 This importance of not rebuilding the image on merge was really impressed upon me by my brilliant colleague Chris Johnson while I was at CoverMyMeds
This model was developed for
Trunk-Based Development
of short-lived feature branches off of the default (main) branch
where that default branch matches production.
This model assumes two separate systems are involved in the full CI/CD process:
- The source code repository (triggered) system which does the Continuous Integration of building, vetting, and promoting the images
- The deployment system which deploys the candidate image to production (ideally) in a blue-green or canary deployment which would trigger the source code repository system to merge the branch and promote the image to its production image repository
Although it may appear more complex than having a single system, having two separate systems has its advantages.
The two different systems offer the benefits of the separation of concerns - code repository synchronization and image building are very different from deployment. It also is more modular allowing one to change without having to change the other (for instance moving to Kubernetes).
Having a separate triggered deployment system also allows for any manual stages in the deployment process, for example any manual reviews or exploratory testing.
In all, having the two separate systems gives the flexibility to add image-based Continuous Integration and integrate it with an existing image-based deployment.
Finally, many Software as a Service (SaaS) source code repository providers such as GitHub and GitLab offer CI/GitOps services. The source code repository (triggered) CI system in this model and presented as A Specific Design example is actually implemented in GitHub Actions using the free subscription.
This model's design intentionally follows a set of values and principles and recognizes and supports certain realities of software development.
This model was designed to meet and uphold these values...
-
Certainty - ensure any changes are known, intentional, and minimized
-
Efficiency - avoid any unnecessary actions and provide the fastest feedback
-
Adaptability - change easily and incrementally to meet different and changing needs
Given the values, this model was designed with these principles...
-
The image is the artifact - fully leverage and realize the certainty of containerization by making the built image the object of the Software Development Lifecycle (SDLC) instead of the raw source code
-
Trunk-Based Development with Continuous Delivery - although it may be possible to adapt this to other branching strategies (e.g. release or development branches), this model follows the efficiency of continuous delivery and deployment with short-lived feature branches
-
As previously discussed, Continuous Integration and Deployment are different and separate
-
Merging to the default (e.g.
main) branch is done after (canary/blue-green) deployment and then image is deployed to production as Continuous Deployment -
Support for multiple types of images
- Deploy Image - the potential production candidate
- Development (Dev) Environment Image - adapt to include a development environment image to use as a portable containerized dev environment and/or to use to perform vetting when deploy image can not be used
-
Support for multiple platform/architecture images, for example
linux/amd64for CI and production environment andlinux/arm64for local development on Apple Silicon -
Idempotent operations to ensure certainty that an image for a branch commit is built and pushed only once as CI runs sometimes just fail
This design recognizes and hopes to support these realities of software development...
-
The Deploy Image may not always pass vetting and this may be intentional for triage and development
-
Deployment of an unvetted Deploy Image may be needed for triage and development, for example in a test environment
-
Only vetted deploy images can be deployed to production for quality and security
Here is a specific design (and implementation) using this model.
This specific design had the constraint that it must build and use a development environment image to vet the commit candidate since the deployment image could not be used as it was immutable and did not have the necessary libraries/packages.
In this specific design, the decision was made that to simplify CI system logic and increase speed and parallelism, an unvetted deploy image will always be built and pushed and then if all applicable vetting steps pass, this unvetted image is promoted to a vetted image repository.
The ramifications of the constraint and decision are...
- Using the dev image for vetting means volume-mounted source code versus any immutable source code in the image
- There may be 2 identical but different-named deploy images if vetting passes
- It is non-deterministic if there is one or two images or if there is a vetted image
- Always pushing an unvetted image means that it is available faster than a vetted image
This is the design of the CI portion of the model and does not include the deployment system design. A separate deployment system can be used to deploy into a test environment or as a blue-green or canary deployment.
When deemed ready for production the deployment system could then use a webhook to trigger the merge operation on the CI system.
😴 In the particular use case for this specific design, there really was no deployment environment since it was for Selenium browser test automation projects
In the CI system there are two separate triggered events and workflows:
-
When a commit is pushed on a code review (Pull Request) branch, the branch-based, commit-tagged images (dev environment and deploy) are built, pushed, and if vetting passes, the unvetted deploy image is promoted to vetted
-
When the branch is merged, the approved images (last commit of branch) are promoted to the production repository to be (re)deployed to production
This diagram illustrates the CI workflow steps and their dependencies when a commit is pushed to the code repository for a branch that is under code review (e.g. Pull Request).
Here the branch-based, commit-tagged deploy and dev environment images are built, pushed, and vetted and if the vetting passes, the unvetted deploy image is pushed to the vetted repository.
Again, in order for a workflow step to be run, all of its dependent steps must pass.
graph TD;
A([Pull Request Commit]) --> B[Build and Push Dev Environment Image];
B --> D[Vet: Code Quality]
B --> E[Vet: Security]
B --> F[Vet: Tests]
B --> G[Vet: Dev Environment Acceptance Tests]
A --> C[Build and Push Unvetted Deploy Image];
C --> H[Vet: Deploy Acceptance Tests]
D --> I[Promote Unvetted Deploy Image as Vetted]
E --> I
F --> I
H --> I
I --> J([Ready for Deployment])
This diagram illustrates the CI workflow steps and their dependencies when the branch that is under code review is merged to the default branch.
Here the images associated with the last commit of the branch (and not the merge commit) are promoted to the production repository.
graph TD;
A([Merge]) --> B[Promote Vetted Deploy Image as Commit-Tagged Production Image];
A --> C[Promote Dev Environment Image as Commit-Tagged Production Dev Image];
B --> D[Promote Commit-Tagged Production Image as latest]
C --> E[Promote Commit-Tagged Production Dev Image as latest]
D --> F([Redeploy Production Image])
This specific CI design is actually implemented in GitHub Actions using the Docker Hub image registry in the projects brianjbayer/sample-login-capybara-rspec and brianjbayer/sample-login-watir-cucumber (all the files are identical) using the reusable GitHub Actions from brianjbayer/actions-image-cicd.
Specifically see the...
-
CI On Push Commit to Code Review Branch implementation at https://github.com/brianjbayer/sample-login-capybara-rspec/blob/main/.github/workflows/on_pr_build_push_vet_images.yml
-
CI On Merge of Code Review Branch implementation at https://github.com/brianjbayer/sample-login-capybara-rspec/blob/main/.github/workflows/on_push_to_main_promote_to_prod.yml
As the specific design shows, this model can be adapted. Other possible adaptations include...
-
Adding image digest tracking and usage for the branch-based, commit-tagged images instead of simply using the commit-tag to increase certainty
-
Using increasing security levels for the image repositories as they are promoted e.g. vetted and production image repositories (registries) are more secure than unvetted
This image-based CI/CD model is proven to be able to be implemented and meets the design values of certainty, efficiency, and adaptability.
It is certain through its image-based nature and idempotent operations of builds and pushes/promotions and in security if using tiered-access-level image registries.
It is efficient by its build-once and promote approach, idempotent operations, workflow step dependencies, and parallelism.
Finally it is adaptable with its separation of CI and deployment, ability to support multiple and/or multi-platform images, flexibility to define the specific workflows to specific needs, and expandability to add on additional certainty and security measures.