First, we need to initialize a traditional policy set. Let’s start with masterfiles from the 3.18.3 release. This way we can go through the process of migrating to the same version with cfbs. We need to download it and unpack it into the root of our repository.
exec 2>&1
MPF_TARBALL_URL=$(cf-remote --version 3.18.3 list masterfiles | tail -n 1)
MPF_TARBALL_FILENAME=$(basename $MPF_TARBALL_URL)
curl -s -O $MPF_TARBALL_URL
tar -zxf $MPF_TARBALL_FILENAME --strip-components=2
rm -f $MPF_TARBALL_FILENAME
ls
:We can see it’s a standard stock policy set.
Next we need to lay down the typical .gitignore for a policy set.
exec 2>&1
printf "# These files are generated and should not become part of the policy set\n" > .gitignore
printf "cf_promises_validated\n" >> .gitignore
printf "cf_promises_release_id\n" >> .gitignore
:Let’s go ahead and initialize the git repository and commit the current changes.
exec 2>&1
git init
git add .
git commit -m "In the beginning there was darkness, then Nick said let there be a stock 3.18.3 policy set" | grep -v "create mode"
:
Now we have a stock 3.18.3 policy set.
Next we need to add some custom policy. Let’s integrate policy in a few different ways to be representative of what you might find in a policy set that has been around for some time.
Long, long ago it was standard practice to modify the default policy to add custom policy to inputs and the bundlesequence. Let’s create a custom policy file and add it to inputs and bundlesequence.
exec 2>&1
cat << EOF > custom-policy-1.cf
bundle agent custom_policy_1
{
reports: 'Policy from \$(this.bundle)';
}
EOF
sed -i '/^\s*"services\/main.cf",/a "custom-policy-1.cf",' promises.cf
sed -i '/^\s*@(def.bundlesequence_end),/a custom_policy_1,' promises.cf
:These days we have Augments (def.json) that allow us to define variables and classes very early during the bundlesequence and the MPF is instrumented with many variables which can be leveraged to influence the behavior of the policy. Let’s add a couple more policies integrated explicitly via augments as well as via autorun.
exec 2>&1
cat << EOF > services/custom-policy-2.cf
bundle agent custom_policy_2
{
reports: 'Hello from \$(this.bundle) \$(with)'
with => join( ",", callstack_promisers() );
}
EOF
cat << EOF > services/autorun/custom-policy-3.cf
bundle agent custom_policy_3
{
meta: "tags" slist => { "autorun" };
reports: 'Hello from \$(this.bundle)';
}
EOF
cat << EOF > def.json
{
"inputs": [ "services/custom-policy-2.cf" ],
"classes": {
"services_autorun": [ "any::" ]
},
"vars": {
"control_common_bundlesequence_end": [ "custom_policy_2" ]
}
}
EOF
#git commit -m "Added custom policy integrated via Augments"
:services/main.cf also commonly contains custom policy, typically methods type promises to call other custom policy.
exec 2>&1
sed -i '/^\s*# Activate your custom policies here/a "Calling custom_policy_2 from main" usebundle => custom_policy_2;' services/main.cf
:With custom policy in place, let’s commit.
exec 2>&1
git add .
git commit -m "Added custom policy integrated in various ways"
:Now, let’s initialize this repository as a cfbs project.
The easiest way to get a cfbs project started is to use cfbs init.
exec 2>&1
cfbs init --non-interactive
:
By default it will use the most recent version of masterfiles, but we want to first get cfbs replicating our existing policy set so we will remove masterfiles.
exec 2>&1
cfbs remove masterfiles --non-interactive
:And add it back at a specific version.
exec 2>&1
cfbs add masterfiles@3.18.3 --non-interactive
:At this point we can cfbs build.
exec 2>&1
cfbs build
:
Now we can see how our cfbs built 3.18.3 policy differs from our current policy set. We recursively diff the root of our policy set against the cfbs build
result in out/masterfiles.
exec 2>&1
diff --exclude=out --recursive --unified . out/masterfiles
:
We can see there are a few differences, but first let’s focus on the files that are missing from the cfbs built policy.
exec 2>&1
diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \."
:
We can ignore the .git, .gitignore, and cfbs.json.
exec 2>&1
diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \." | grep -vP ".git|.gitignore|cfbs.json"
:
Now we can easily see that we need to get custom-policy-1.cf, def.json, services/autorun/custom-policy-3.cf, and services/custom-policy-2.cf into the cfbs built policy set. Let’s cfbs add them.
exec 2>&1
cfbs add custom-policy-1.cf --non-interactive
:Let’s take a look at what cfbs add did.
exec 2>&1
git log -p -n 1
:
We can see it added a new JSON object with steps:
copy ./custom-policy-1.cf services/cfbs/custom-policy-1.cf- This copies
custom-policy-1.cftoservices/cfbs/custom-policy-1.cfrelative to the root of the built policy. policy_files services/cfbs/custom-policy-1.cf- This adds
services/cfbs/custom-policy-1.cftoinputsindef.jsonso that the policy file is parsed. bundles custom_policy_1- This adds
custom_policy_1todefault:def.control_common_bundlesequence_endindef.jsonso that the bundle will be run as part of the bundlesequence.
At least for now, we are looking to replicate the existing policy set while trying to avoid modifications to the MPF, so let’s edit cfbs.json so that custom-policy-1.cf is placed in the root of the built policy.
exec 2>&1
sed -i 's|services/cfbs/custom-policy-1.cf|custom-policy-1.cf|g' cfbs.json
:Let’s check that the build steps for custom-policy-1.cf are as desired.
exec 2>&1
git diff
:
That looks correct, so we can commit those changes to the cfbs project.
exec 2>&1
git add cfbs.json
git commit -m "Fixed custom-policy-1.cf build integration target location, inputs and bundlesequence"
:Next we do similar with the other custom policy files, first services/custom-policy-2.cf.
First we add the local policy to the project.
exec 2>&1
cfbs add services/custom-policy-2.cf --non-interactive
:We see that it’s target location is not as desired.
exec 2>&1
git log -p -n 1
:
We correct the target location.
exec 2>&1
sed -i 's|services/cfbs/services/custom-policy-2.cf|services/custom-policy-2.cf|g' cfbs.json
:We inspect our change.
exec 2>&1
git diff
:
We commit our change fixing the target location for custom-policy-2.cf
exec 2>&1
git add cfbs.json
git commit -m "Fixed custom-policy-2.cf build integration target location, inputs and bundlesequence"
:Finally, we start integrating the missing custom-policy-3.cf.
exec 2>&1
cfbs add services/autorun/custom-policy-3.cf --non-interactive
:We again see the target is not as desired, but also we see that we don’t need to explicitly include this policy file in inputs or the bundlesequence as it leverages autorun for inclusion.
exec 2>&1
git log -p -n 1
:
We adjust the target path and commit the changes to the cfbs project, and we redact the addition of the file to inputs (since it’s in the services/autorun directory as well as it’s inclusion to the top level bundlesequence.
exec 2>&1
# Fix the target path for custom-policy-3
sed -i 's|services/cfbs/services/autorun/custom-policy-3.cf|services/autorun/custom-policy-3.cf|g' cfbs.json
# Remove custom-policy-3.cf from inputs and bundlesequnce
sed -i '/.*policy_files services\/autorun\/custom-policy-3.cf.*/d' cfbs.json
sed -i '/.*bundles custom_policy_3.*/d' cfbs.json
# Now we need to correct the JSON since the copy is the only build step and the
# last entry of a JSON array must not contain a comma
sed -ri 's/("copy.*)custom-policy-3.cf",/\1custom-policy-3.cf"/g' cfbs.json
:exec 2>&1
git diff
:
exec 2>&1
git add cfbs.json
git commit -m "Fixed custom-policy-3.cf build integration target location, redacted inputs and bundlesequence"
:Finally we can see that the policy set resulting from cfbs build is not missing anything.
exec 2>&1
cfbs build
diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in \." | grep -vP ".git|.gitignore|cfbs.json"
:
Let’s look at the files in the cfbs built result that are not in the current policy.
exec 2>&1
diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in out"
:
We have some extra files, let’s take care of those by editing cfbs.json and deleting them in the build steps of the masterfiles module.
exec 2>&1
# Modify masterfiles build steps to remove files not delivered by release tarball
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete cfe_internal\/core\/watchdog\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete cfe_internal\/enterprise\/ha\/ha_info.json",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete .github",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete inventory\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete lib\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete LICENSE",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete modules\/promises",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete .no-distrib",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete services\/autorun\/README.md",\n\1\2/' cfbs.json
sed -ri 's/(.*)("copy .\/ .\/")/\1"delete templates\/README.md",\n\1\2/' cfbs.json
:After making the edits to the masterfiles module build steps we can review our changes.
exec 2>&1
git diff
:
And we can make sure that the result of cfbs build is not missing any files.
exec 2>&1
cfbs build
diff --exclude=out --recursive --unified . out/masterfiles | grep "Only in out"
:
Now all that remains is reviewing any other differences between our traditionally managed policy with the result of cfbs build.
exec 2>&1
diff --exclude=out --recursive --unified . out/masterfiles
:
The diff above highlights differences in def.json, .git, .gitignore, promises.cf, and services/main.cf.
- The difference in the
inputskey and thecontrol_common_bundlesequence_endarray in thevarskey is expected as we migrated a modification includingcustom-policy-1.cfin inputs and bundlesquence frompromises.cftodef.jsonvia the build steps for that module, removing an unnecessary modification to the vendored files which is also reflected in the differences ofpromises.cf - We are lacking the class definition for
services_autorunto enable the autorun service. - We are lacking the modification to
services/main.cfexplicitly running thecustom_policy_2bundle.
We can address services/main.cf as we have addressed other files deployed from our traditionally managed policy set, first adding the file.
exec 2>&1
cfbs add services/main.cf --non-interactive
:Then adjusting the build steps as necessary.
exec 2>&1
# Overwrite the vendored services/main
sed -i 's|services/cfbs/services/main.cf|services/main.cf|g' cfbs.json
# Remove build steps for services/main.cf for inputs and bundlesequnce as it's
# already inclided by default in the framework as a vendored file
sed -i '/.*policy_files services\/main.*/d' cfbs.json
sed -i '/.*bundles main.*/d' cfbs.json
# Now we need to correct the JSON since the copy is the only build step and the
# last entry of a JSON array must not contain a comma
sed -ri 's/("copy.*)main.cf",/\1main.cf"/g' cfbs.json
:We can address enabling autorun by simply adding in the autorun module.
exec 2>&1
cfbs add autorun --non-interactive
:This leaves only the expected differences between the traditionally manged policy set and the cfbs built policy set.
exec 2>&1
cfbs build
diff --exclude=out --recursive --unified . out/masterfiles
:
At this point we have completed the process of bringing our policy set under cfbs management. Look around build.cfengine.com for modules that interest you, try adding and removing modules, you can even try cfbs update masterfiles to see how the policy framework upgrade has been simplified.
If you have ideas for improving the feature set or behavior of cfbs please open a ticket on the issue tracker.