Puppet is a tool to enable organizations to deliver change. This document describes the workflow and process for doing that. Breaking it down, there are three parts to this document.
- Overview all the different kinds of infrastructure-as-code content users and developers can interact with.
- Describe the development process for proposing changes to any of the Puppet content described above.
- Describe the deployment process for accepting, promoting, and eventually deploying in production changes to any of the Puppet content described above.
The bulk of Puppet content developers will interact with is infrastructure-as-code, versioned and committed to a Git version-control repository.
When using CD4PE, every Git reponsitory, for any kind of content, works the same way. If you search for Git 101, the top tutorials that you find will describe how to interact with all standard Git repositories, such as these. There will typically be a master branch which tracks the latest developing code, tags to identify and name meaningful versions, and other branches coming and going as part of the code development lifecycle.
CD4PE (and Puppet) use a small set of Git branches for bookkeeping, which are not part of the standard Git process. These branches will be visible to users, but should be protected so that users cannot modify them. CD4PE will manage these bookkeeping branches automatically.
The control-repo is the hub for all Puppet configuration content. In its purest form that's all it is. More typically though, the control-repo both acts as a hub for modularized content, and also embeds some content directly. Typically, the embedded content includes the "role" module, "profile" module, as well as Hiera data.
In a pure control-repo, even that content will be modularized rather than embedded.
This is the switchboard or ledger that makes the control-repo a hub. It is a file that enumerates all of the modular content Puppet should incorporate and make available.
Reading the Puppetfile should convey a good sense of all the Puppet content being used at a given site or Puppet implementation.
Depending on how much content has been embedded in the control-repo, there will be a small handful of Puppet configuration files present, and possibly some directories for the embedded content.
A Puppet module is a collection of content laid out in a particular way. A module can contain desired state code, Bolt tasks or plans, Puppet extensions, or a variety of other things. The module is the standard content container for everything Puppet.
Puppet knows how to consume modules automatically, provided they are placed somewhere in its modulepath.
Some Puppet content may be useful to partition off for development purposes, even if it is not technically a Puppet module. The prime example of this is Hiera data. Hiera data may be kept in a dedicated Git repository, and attached to the control-repo using the Puppetfile, just like any standard Puppet content would be.
There exist other use cases for carving off non-module Git repositories but they are rare. The commonality is that all such content will be committed to a Git repository and be referenced in the control-repo's Puppetfile.
Puppet typically doesn't consume non-module content automatically, and correct consumption of this content requires some kind of configuration in one of the control-repo's configuration files.
There is a single common high-level process for developing a change, regardless of what kind of content the change relates to. The process is the same whether the content is in the control-repo, a module, or a non-module repo. The change development process is driven through Git.
A change to content in Git is considered finished once the change has been merged into the Git master branch. Note that a finished change is not the same as a deployed change; a finished change is a change ready to be tested and deployed using the deployment process described below.
The high level process for developing a change is:
- Create a feature branch in Git, based off of
master, in which to develop your change. - Work on your change. Edit files, run tests, and do anything else you need to in order to realize, in code, the change you want to make. (Note: The details of what files you edit and how you test your change locally as a developer will vary, depending on which kind of content you're working on).
- Git commit all the files you've changed to your feature branch. You may be committing frequently as you iterate, or you may not commit until you're fully satisfied with your work. How much you like to involve (or not involve) Git in your process up to this point is totally up to you.
- Submit a pull/merge request to have your feature branch merged into
master. - Assuming whatever CI or process requirements your organization uses are satisfied, you or someone else can merge your feature branch into
master, resulting in your change being available and ready to be put on the path to production.
When you submit a pull/merge request against a Puppet content repo, the Continuous Integration pipeline configured in CD4PE for that repo will run and report back success/failure results in your pull/merge request.
Pipelines in CD4PE are not cleanly separated into CI and CD. Instead, you see a single contiguous pipeline. However, it is a fair approximation to say that everything in a CD4PE pipeline that occurs before a special marker called the Pull Request Gate is the CI portion of the pipeline, and is probably focused on validating that proposed changes pass requisite acceptance tests or governance requirements.
The CI portion of CD4PE pipelines therefore aids the development workflow.
Once a proposed change has been merged into the master branch of a Puppet content repo, the new content version (in the master branch) containing the change needs to be deployed.
Abstractly, to deploy a change means to select a version of content, select a target environment, and cause the selected change to take effect in the selected environment.
In CD4PE, this means
- Pick a version.
- Pick a target environment.
- Optionally, set some deployment parameters.
- Press the "deploy" button.
A version is any commit in the history of the master branch. There are no chronological requirements for picking a version to deploy. Deploying a version will neither change the master branch nor effect it in any way.
An environment is a target set of nodes to which content versions may be deployed. Environments are configured using Environment Node Groups in the PE console.
Per the definition of deploying (given above), environment node groups need to be configured as a prerequisite to deploying any content versions. At a minimum, the environment you want to target needs to have been defined as an environment node group.
Some basic characteristics of defining environment node groups:
- Nodes can only belong to a single environment node group
- The environment node group may be named anything that makes sense to your organization
- The environment node group must be assigned a "Puppet environment".
- The functional importance of a "Puppet environment" relates to how CD4PE will perform bookkeeping of the environment's content in Git (using the bookkeeping branches mentioned earlier)
- Each environment node group should be assigned a unique "Puppet environment" value. This ensures simple bookkeeping and that change deployments will not affect nodes they are not supposed to.
The base process:
- Pick a version.
- Pick a target environment.
- Optionally, set some deployment parameters.
- Press the "deploy" button.
A special note about the control-repo is that because it contains the Puppetfile, which is the switchboard or ledger to which all modularized content is attached, how that modularized content is deployed in a target environment depends on the Puppetfile deployed to that environment.
Modularized content can be attached to a Puppetfile statically, such that the modularized content cannot be independently deployed or updated, or the content can be attached dynamically, so that the modularized content can be deployed independent of the control-repo using a different CD4PE pipeline or deployment actions.
If modularized content is defined statically in the Puppetfile, the only way to update it is to develop a change to the control-repo itself which modifies the Puppetfile to update the static module, and then deploy that change to the control-repo.
The base process:
- Pick a version.
- Pick a target environment.
- Optionally, set some deployment parameters.
- Press the "deploy" button.
The inverse of the special note about control-repos above is that to be able to deploy module changes this way, the control-repo version deployed to the target environment must have a Puppetfile that attaches this modularized content dynamically. If the target environment does not have such a Puppetfile, it will not be possible to independently deploy the module changes.
Many organizations have a semi-linear sequence of target environments to which a change will be deployed—a sequence which terminates at the highest-value target environment. Oftentimes this final target is called "production".
A pipeline in CD4PE is automation reflecting this sequence of deployments. It is a progression of deployments (and other actions) which when triggered, will run in a defined manner, pausing, terminating, or continuing after each step as configured to by the pipeline creator.
The pipeline can contain many kinds of actions besides deployments too. As mentioned in the "Developing" section, the first part of a CD4PE pipeline up until the special Pull Request Gate step is typically CI validation and testing actions, rather than deployment actions. Deployment actions will usually not be placed in the pipeline until after this step.
The continuous element of a pipeline is in reference to what triggers it, or when it is run. Pipelines typically start automatically as soon as a new change has been merged to the master branch of a content repository.
Deployment pipelines will typically deploy validated changes automatically up to some level lower than "production", but pause or wait for human approval before continuing to run and deploy to more sensitive higher tiered environments.