This gist has files that are used to automate chef repository submodules used for roles, data bags, cookbooks and environments, ultimately uploading the maintained chef repository to chef via knife using Jenkins and Github.

0-auto-manged-chef-repo.md

The files in this gist are for having Jenkins automatically manage a chef repository using git submodules. This allows for clean, clutter free management of individual cookbooks, and individual respositories for roles, environments and data bags in our chef-repo.

The process relies on using Github (we use Github Enterprise) and Jenkins in combination with the Jenkins Github plugin to notify Jenkins when a repository has changed.

Our chef-repo directory looks something like:

chef-repo
    - cookbooks
          - Each cookbook is a git submodule managed by Jenkins
    - data_bags (git submodule managed by Jenkins)
    - environments (git submodule managed by Jenkins)
    - roles (git submodule managed by Jenkins)
    - All other normal files and directories

So that we can use git submodules for dependency cookbooks that are maintained by a third parties, there is a dependencies.json file placed in the root of each cookbook. See 0-dependencies.json and 0-dependencies-json-notes.md.

Each cookbook has a Jenkins job that recives a Github callback when changes are pushed into it. These jobs all have two build steps:

• 1-cookbook-update.sh
• 2-cookbook-dependency.py
• 3-chef-repo-push.sh

1-cookbook-update.sh updates the chef repo for the updated cookbook, while 2-cookbook-dependency.py adds or updates cookbook dependencies. Note that at this time these scripts do not do conflict resolution for different versions or repositories of the same dependency across cookbooks. 3-chef-repo-push.sh pushes the submodule updates back to Github.

4-other-build.sh is used for each respository that contains the roles, environments, and data bags. Roles, environments and data bags each have a Jenkins job which like the cookbooks will update the submodule reference in the chef-repo repository.

When chef-repo is updated, Github makes a callback to Jenkins which triggers a job to upload changes to the Chef server. All of the scripts use a uniform commit message style to chef-repo so that the 5-knife_production_upload.sh script can extract the proper type of update to run and what cookbook(s) to update.

If you want to make sure that cookbooks pass tests prior to inclusion, tt should be trivial to add build-steps in Jenkins that execute unit tests or other acceptance testing prior to making the commits to chef-repo.

0-dependencies-json-notes.md

The dependencies.json file allows for more granularity than metadata.rb, specifying the remote repository and optionally the branch and revision to use. It should be placed in the root directory for each cookbook you create, but can be empty:

{}

0-dependencies.json

{
    "cookbooks": {
        "build-essential": {
            "url": "git://github.com/opscode-cookbooks/build-essential.git",
            "branch": "master",
            "revision": "2991fc6e5448e5b2378eb6882e9c1ce557d828f0"
        },
        "cron": {
            "url": "git://github.com/opscode-cookbooks/cron.git"
        },
        "hp_server": {
            "url": "git://github.meetmecorp.com/chef/hp_server.git"
        },
        "ohai": {
            "url": "git://github.com/opscode-cookbooks/ohai.git"
        },
        "python": {
            "url": "git://github.com/opscode-cookbooks/python.git"
        },
        "resolver": {
            "url": "git://github.com/opscode-cookbooks/resolver.git"
        },
        "sudo": {
            "url": "git://github.com/opscode-cookbooks/sudo.git"
        },
        "supervisor": {
            "url": "git://github.com/opscode-cookbooks/supervisor.git"
        },
        "yum": {
            "url": "git://github.com/opscode-cookbooks/yum.git"
        }
    }
}

0-knife.rb

# Goes in /var/lib/jenkins/.chef/
log_level :info
log_location STDOUT

syntax_check_cache_path  "#{ENV['HOME']}/.chef/syntax_check_cache"
chef_server_url "#{ENV['KNIFE_CHEF_SERVER']}"
client_key "#{ENV['KNIFE_CLIENT_KEY']}"
node_name "#{ENV['KNIFE_NODE_NAME']}"
validation_client_name "#{ENV['KNIFE_VALIDATION_CLIENT_NAME']}"
validation_key "#{ENV['KNIFE_VALIDATION_CLIENT_KEY']}"
cookbook_path "#{ENV['KNIFE_COOKBOOK_PATH']}"

1-cookbook-update.sh

# This is the first build step for a cookbook repo update which is 
# triggered by a GitHub callback to Jenkins on commit

# Reset the workspace, we don't want the git checkout of $JOB_NAME
cd $WORKSPACE/..
rm -rf workspace
mkdir workspace
cd workspace

# Pull down repo
git clone git@github.com:YOUR/chef-repo.git
cd repo

# Add or update the cookbook
if [ ! -d "cookbooks/${JOB_NAME}" ]; then
    git submodule add $GIT_URL cookbooks/$JOB_NAME
    git submodule init cookbooks/$JOB_NAME
    git commit -m "Adding cookbook ${JOB_NAME}" .gitmodules cookbooks/$JOB_NAME
else
    git submodule update --init cookbooks/$JOB_NAME
    cd cookbooks/$JOB_NAME
    git pull -f -u origin master
    cd ../..
    git commit -m "Updating cookbook ${JOB_NAME} to ${GIT_COMMIT}" .gitmodules cookbooks/$JOB_NAME
fi

2-cookbook-dependency.py

import json
import os
import subprocess
import sys

base_path = os.environ['WORKSPACE'] + '/' + 'repo'

cookbook = os.environ['JOB_NAME']
cookbook_path = '%s/cookbooks/%s' % (base_path, cookbook)
dependencies = '%s/dependencies.json' % cookbook_path


def execute(command, ignorable=False):
    p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = p.communicate()
    if p.returncode:
        print 'Command failure [%s] returned %s' % (' '.join(command), 
                                                    p.returncode)
        if ignorable:
            return False
        print
        print stderr
        print
        sys.exit(p.returncode)
    return stdout

os.chdir(base_path)
if os.path.exists(dependencies):
    with open(dependencies, 'r') as handle:
        data = json.load(handle)

    cookbooks = data.get('cookbooks', dict())
    for name in cookbooks:
        print 'Processing dependency cookbook %s' % name
        dependency_path = '%s/cookbooks/%s' % (base_path, name)
        if not os.path.exists(dependency_path):
            print 'Adding dependency cookbook %s' % name
            execute(['git', 'submodule', 'add', cookbooks[name]['url'], 'cookbooks/%s' % name])
            print 'Committing addition of dependency cookbook %s' % name
            execute(['git', 'commit', '-m',  'Adding dependency cookbook %s' % name, 
                     '.gitmodules', 'cookbooks/%s' % name])

        print 'Updating dependency cookbook %s' % name
        execute(['git', 'submodule', 'update', '--init', 'cookbooks/%s' % name])

        os.chdir(dependency_path)            
        if cookbooks[name].get('branch'):
            if not  execute(['git', 'checkout', '-b', cookbooks[name].get('branch', 'master')], True):
                execute(['git', 'checkout', cookbooks[name].get('branch', 'master')])

        execute(['git', 'pull', '-f', '-u', 'origin', cookbooks[name].get('branch', 'master')])
        os.chdir(base_path)
              
        if 'revision' in cookbooks[name]:
            # Change to that revision
            os.chdir(dependency_path)
            execute(['git', 'reset', '--hard', str(cookbooks[name]['revision'])])   

        # Commit the specific revision for that cookbook
        os.chdir(base_path)
        stdout = execute(['git', 'status', '--porcelain'])
        if stdout:
            print 'Committing updates to dependency cookbook "%s"' % name
            revision = cookbooks[name].get('revision', 'HEAD')
            execute(['git', 'commit', '-m',  'Updating dependency cookbook %s to %s' % (name, revision),
                     '.gitmodules', 'cookbooks/%s' % name])
        else:
            print 'No changes to commit for dependency cookbook %s' % name

3-chef-repo-push.sh

# This is the third buildstep for a cookbook
cd $WORKSPACE/repo
git push origin master

4-other-build.sh

# This is the only build step for updates to the data bag, role and 
# environment repositories which are triggered by a GitHub callback 
# to Jenkins on commit

# Reset the workspace, we don't want the git checkout of $JOB_NAME
cd $WORKSPACE/..
rm -rf workspace
mkdir workspace
cd workspace

# Pull down repo
git clone git@github.com:YOUR/chef-repo.git
cd repo

# Update the $JOB_NAME
git submodule update --init $JOB_NAME
cd $JOB_NAME
git pull -f -u origin master
cd ..
git commit -m "Updating ${JOB_NAME} to ${GIT_COMMIT}" .gitmodules $JOB_NAME

# Push the change
git push origin master

5-knife_production_upload.sh

# Used for the job that is notified when the master chef repository is updated when
# a cookbook, role, environment or data bag repository updates chef-repo. When 
# cookbook-dependency.py, cookbook-update.sh or other-buid.sh are run, they will 
# update chef-repo and push the changes, causing GitHub to trigger Jenkins to run 
# the job with this shell script as its build step.

# Update the submodules to the proper revisions
git submodule update --init --recursive

# Knife configuration
export KNIFE_CHEF_SERVER="YOUR_CHEF_SERVER"
export KNIFE_CLIENT_KEY="$HOME/.chef/client.pem"
export KNIFE_NODE_NAME="jenkins"
export KNIFE_VALIDATION_CLIENT_NAME="chef-validator"
export KNIFE_VALIDATION_CLIENT_KEY="$HOME/.chef/validation.pem"
export KNIFE_COOKBOOK_PATH=$WORKSPACE/cookbooks

COOKBOOKS=()
DEPENDENCY_COOKBOOKS=()
GITLOG=`git log -n 100  --pretty=oneline`
set -f; IFS=$'\n'; for LINE in $GITLOG; do
    if [ "${LINE:0:40}" == "$GIT_PREVIOUS_COMMIT" ]; then
        echo "Exiting on changeset ${LINE:0:40}";
        break
    fi

    OBJECT=`echo $LINE | perl -n -e '/^(\w{40})\s+(Adding|Updating)\s(cookbook|dependency\scookbook|environments|data_bags|roles)/ && {print "$3"}'`

    if [ "$OBJECT" == "cookbook" ]; then
        COOKBOOK=`echo $LINE | perl -n -e '/^\w{40}\s+(Adding|Updating)\scookbook\s([\w-_]+)(\sto|).*$/m && {print "$2\n"}'`
        COOKBOOKS+=("$COOKBOOK")
    
    elif [ "$OBJECT" == "dependency cookbook" ]; then
        COOKBOOK=`echo $LINE | perl -n -e '/^\w{40}\s+(Adding|Updating)\sdependency\scookbook\s([\w-]+)\s(to.|)*$/m && {print "$2\n"}'`
        DEPENDENCY_COOKBOOKS+=("$COOKBOOK")

        elif [ "$OBJECT" == "data_bags" ]; then
        cd $WORKSPACE/data_bags
        FILES=`git diff-tree --no-commit-id --name-only -r $(git log --oneline -n 1 | awk '{print $1}')`
        cd $WORKSPACE
        for data_bag in ${FILES[@]}; do
            knife data bag from file ${data_bag/\// }
        done

    elif [ "$OBJECT" == "environments" ]; then
        cd $WORKSPACE/environments
        FILES=`git diff-tree --no-commit-id --name-only -r $(git log --oneline -n 1 | awk '{print $1}')`
        cd $WORKSPACE
        for environment in ${FILES[@]}; do
            knife environment from file $environment
        done

    elif [ "$OBJECT" == "roles" ]; then
        cd $WORKSPACE/roles
        FILES=`git diff-tree --no-commit-id --name-only -r $(git log --oneline -n 1 | awk '{print $1}')`
        cd $WORKSPACE
        for role in ${FILES[@]}; do
            knife role from file $role
        done
    fi

done

UPLOAD=($(printf "%s\n" "${DEPENDENCY_COOKBOOKS[@]}" | sort -u ))
for COOKBOOK in ${UPLOAD[@]}; do
    knife cookbook upload $COOKBOOK
done
UPLOAD=($(printf "%s\n" "${COOKBOOKS[@]}" | sort -u ))
for COOKBOOK in ${UPLOAD[@]}; do
    knife cookbook upload $COOKBOOK
done