snapshot of our git workflow

README.md

Software Defined Architecture

Goals

These manifests are used to more effectively manage ISE's infrastructure. Some reasons for this include:

• Reproducible: If we lose a host or need to rebuild, Puppet enables us to rebuild to a known good state.

• Distributable: With Puppet manifests stored in this repository, it is easier to accept infrastructure help from members of the ISE technical community, without necessarily giving root access out.

• Accountable: By funneling as much infrastructure work through Puppet as possible, we ensure the infrastructure has a very clear audit trail for specific changes.

• Scalable: By moving configuration options to parameters, we can add nodes quickly.

Change management

Puppet does not replace change management. Puppet supports the change management process and provides consistent implementation within the scope of ISE's RFC process. Specifically:

• Every change that impacts Prod MUST have an RFC, regardless of whether the change is applied manually or via Puppet

• Changes that do not impact Prod do not require an RFC

Puppet Overview

ISE uses Puppet to apply consistent, repeatable changes to hosts. Currently, this includes Linux and Windows. In the future, it may also include network devices. Nevertheless, ISE follows a consistent change management process for every change that impacts Prod, regardless of whether changes are applied manually or via Puppet.

The output from Puppet is a compiled catalog that represents a directed acyclic graph (DAG) of resources (configuration items) and is applied to each host via a process referred to at ISE as "host-enforce". Notably, host-enforce is orchestrated by Operations in Prod on-demand to perform a Puppet run.

The input to Puppet is a set of plain-text files written in the Puppet Domain-Specific Language (DSL), combined with flat-file data and referred to as the Source of Truth (SoT). ISE stores the input SoT in Git, a version control system that maintains an immutable history of all changes to all systems.

The Puppet admins ("maintainers") have a documented workflow within the Git repo for Puppet, and it is the detailed implementation. The following is a high-level overview to illustrate the boundaries and responsibilities of the workflow.

ISE has adopted a fork + branch workflow with Git such that changes via Puppet follow established Change Management policy.

Key concepts

• A single production branch contains the cumulative history of all changes for every environment, including Dev, Test, DR, and Prod. The production branch therefore contains the complete configuration of every host as we know it (not just Prod).

  The production branch is immutable; once published, the history within the production branch does not change. The integrity of the history is protected via cryptographic hashes. Immutability provides a strong audit trail of actual changes.

• Each conceptual change is staged within a feature branch; the branch is referred to as a feature branch because its goal is to implement a feature or a change. A feature branch may be rewritten as many times as necessary until it's ready for implementation. Feature branches provide a safe environment in which to develop and test changes.

• A special feature branch called the daily change branch is created if there are approved RFCs on a given day. The daily change branch extends the concept of a feature branch to include tracking via ISE's RFC approval process.

• The maintainers have created a system known as host impersonation that enables ISE to test Prod-related feature branches outside of Prod. Host impersonation enables robust smoke tests.

Key steps from change creation to implementation

1. Verification: Every feature branch must pass a battery of unit tests and smoke tests before it is considered ready for implementation.

   a. A Continuous Integration server automatically runs the unit tests without user intervention. This is the first step of verification. Unit tests must pass before proceeding to the next step.

   b. The change author documents the steps needed to check whether a change is considered successful. We refer to this as the smoke test, and the author must manually perform the smoke test via host-enforce.

   c. The change author adds their sign-off on the branch when the author thinks the branch is ready for implementation.

2. Code review: A feature branch is a collaborative effort across multiple teams and system "owners", so technical peers review all changes during or after the verification steps, but before the approval steps. Optionally, technical peers reproduce the smoke test. Code review is the final step before declaring a feature branch ready for implementation.

3. Approval: An RFC is submitted after a change is considered ready for implementation.

   a. Each RFC is created and assigned to the person in charge of running host-enforce in Prod (i.e., daily change branch).

   b. A maintainer stages approved RFC changes in a daily change branch that is used during a host-enforce to apply those changes to Prod. Non-approved RFC changes remain in a regular feature branch until approved.

   c. A maintainer merges the daily change branch to production after the daily change branch is successfully applied and the immediate system check-out is good.

   d. A maintainer merges non-RFC changes to production branch when proven good.

More detail

If you would like more detail or to author the puppet manifests, please read the documents in this repo:

• CONTRIBUTING.md

• docs/guides/*

• wiki

A.md

• README.md provides overview and key concepts

• CONTRIBUTING.md describes workflow

CONTRIBUTING.md

Contributing

Initial setup

1. Fork the repo
2. Clone the repo to your workstation
3. Change into the repo
4. Run bundle install --path=~/.bundle
5. Run rake ise_setup

Workflow

Topic branch + pull request (PR)

To submit a patch to ISEexchange/puppet, fork the repo and work within a topic branch of your fork.

1. Set up a remote tracking branch

   git checkout -b <branch_name>
   
   # initial push
   git push -u origin <branch_name>

2. Ensure your branches are up-to-date:

   # update your topic branch(es)
   for branch in $(git branch | awk '{print $NF}'); do
     echo ========= $branch ===========
     git checkout $branch && rake rebase || echo failed to rebase
     echo
   done
   git checkout production
   
   # force-push since you rewrote history
   git push --all --prune -f origin

3. Hack away at your branch and run unit tests

   # Edit your branch and commit
   git add <files>
   git commit [--amend]
   
   # Run unit tests
   rake diffspec
   
   # Force push
   git push -f

4. Submit a Pull Request

Autostager

The puppet-autostage script uses the Github REST API v3 to detect pull requests (PR) and automatically stage branches for testing with Puppet.

Your PR is automatically staged or updated when:

• you push more commits
• you rebase, then force-push

Your staged branch is automatically unstaged when:

• you close your pull request
• you delete the remote branch

The autostager sends email notifications:

• verbose diff to a large group when the PR is initially created
• brief reminder twice daily to the same group
• verbose diff to all authors having commits in a particular PR every time the PR is updated with more commits

Diffspec

We use Jenkins to run a few commands against every pull request:

bundle install --path=~/.bundle || bundle update
rake diffspec

You should run those commands, too. The diffspec executes unit tests against every module that has changed in your branch as well as the top-level "common" tests.

WARNING: The unit tests give you a safety net against many mistakes. Ensure your branch is diffspec-clean before running your branch on a live host. If your branch is not clean, you could wipe out a live host, requiring a fresh rebuild to recover.

Our Jenkins host is currently at http://td-lss04.test.ise.com:8080

Smoke-Test

After rake diffspec is clean, you can smoke-test on a live host.

You should receive an email from no-reply@ise.com (see Autostager, below) within 60 seconds of pushing.

Test on a telx host by copy-pasting the commands from the email. The command should be something like:

host-enforce.sh -b branch_name

See the document, {file:TESTING.md}, in this directory. It contains instructions for smoke test, host impersonation, rspec, and the helpful files you can use to validate your code. It also contains a list of custom facter facts that you can use for your puppet manifests.

If your branch is not perfect, amend your commit and force-push.

Sign-off

Sign off on your commit to indicate that you have performed your test plan.

git commit --amend -s
git push -f

Signing off on a patch means that the tester has confirmed the following:

• They have stepped through any available test plan
• They confirm the functionality is working
• They note any anomalies and determine whether or not these block the patch from going into the next stage of QA.

A person signing off on a patch is responsible for making sure that the functionality is working as described in the patch. We know that people miss stuff - no worries! Just be willing to re-do your signoff for any revisions and no one will complain at you.

More information on signed-off-by lines:

• http://goo.gl/QPRRN
• sections 12-14 of http://goo.gl/2JWPA

Merge!

Add a comment to your pull request, saying that you have signed off on the branch and request approval for it. Make sure you mention the maintainers to get their attention.

{include:file:docs/guides/Maintainers.md}

After your branch is merged (or whenever upstream/production is updated), you need to do some repo maintenance. See {file:docs/guides/Git-Housecleaning.md Git-Housecleaning}.

If your change requires an RFC, a maintainer will cherry-pick your commit into the daily change branch.

Linear history

Use either git rebase upstream/production or rake rebase to update your branch.

• The autostager alerts you via email if your branch is "behind" the production branch, in which case you should rebase as described above.
• You must force-push after rebasing
• We try to never force-push to the production branch

The primary reason for this is to maintain a clean, linear history via "fast-forward" merges to the production branch. A clean, linear history in the production branch makes it easier to troubleshoot regressions, {file:docs/guides/Git-Bisect.md bisect history}, and follow the timeline.