Cookbook Autoload RFC

autoload-rfc.md

RFC	Author	Status	Type
unassigned	John Keiser <jkeiser@chef.io>	Draft	Standards Track

Autoloaded Core Resource Cookbooks

Chef core resources will be moved into cookbooks and made implicit dependencies of all user cookbooks, so that cookbook tools can be used to edit and test them, and users can (sometimes) upgrade them without downloading a whole new Chef.

Motivation

As a Chef developer,
I want the core Chef resources in cookbooks,
So that I can use test-kitchen and the other awesome Chef ecosystem tools to test the crap out of them on all platforms.

As a Chef user,
I would like to be able to reference core resources without `depends` or `require`,
So that I have less ceremony to worry about.

As a Chef developer,
I would like to be able to update the set of core resources independently of Chef,
So that I can deliver those changes to users before

As a Chef user,
I would like to be able to version and update Chef core resources by updating cookbooks,
So that I can version and update core resources using tools I understand.

As a Chef user,
I would like to be able to upgrade core resources without upgrading Chef,
So that I can get new core resources without moving to unstable versions or fix bugs locally.    

As a Chef developer,
I would like to decouple resource definitions from Chef into cookbooks,
So that I can get more people to review and fix them.

Specification

Three things will happen:

1. Chef core resources, Cheffish and Chef Provisioning will be published as cookbooks on Supermarket.
2. The core cookbooks will come preinstalled as part of Chef 12.6 and be automatically at the end of the cookbook_path.
3. Chef tools will act as if all cookbooks depend on core cookbooks.

As a result, the core cookbooks will be uploaded to the server as part of policies and berksfiles, automatically loaded as part of Chef runs, and downloaded from the server if (and only if) the user upgrades their cookbooks.

Chef Core: the 'chef-core' cookbook.

The chef-core cookbook will contain all resources and providers from lib/chef/resource and lib/chef/provider. They will be loaded when the Chef run loads libraries from the chef-core cookbook. This will happen before any other cookbook's libraries are loaded because it is implicitly considered the first dependency of any cookbook.

This cookbook will fail to load if run on older versions of Chef with resources already in the core.

Backcompat mitigation

This will not impact any existing Chef cookbooks. However, it can affect programs that use resources outside of a Chef run after calling require 'chef/resources'.

Cheffish and Chef Provisioning cookbooks

Cheffish and Chef Provisioning will be shipped as cookbooks (as well as gems) and loaded after the chef-core cookbook, activating themselves as gems.

Backcompat mitigation

There is a potential conflict if a user has written a program which manually activates a preinstalled cheffish or chef-provisioning gem, and kicks off an in-process Chef run manually. To mitigate this rather unique circumstance, the cheffish and chef-provisioning cookbooks will check if the gem is activated before activating themselves, and raise an error if the activated version is different from the cookbook's version.

Preinstalling Cookbooks in Chef

The Preinstalled Cookbooks RFC handles this, but the key takeways are:

1. The chef-core, cheffish and chef-provisioning cookbooks will be preinstalled as part of the Chef Client and ChefDK gems.
2. The cookbooks will be a part of the cookbook cache in chef-client, so that if users download or depend on a newer version of core cookbooks, they will be picked up first.
3. If a user cookbook depends on specific versions of chef-core, cheffish or chef-provisioning, those will be downloaded from the server as normal.

Chef Tools And Dependencies

These common tools will obey the above rules. The changes made look like:

Chef Client

The cookbook selection, download and load process will be modified so that

(a) preinstalled cookbooks are included in the cookbook cache (b) preinstalled cookbooks are prepended to the node's depsolved cookbook list if they are not already there (c) preinstalled cookbooks are synchronized with the server if the server has them, but the server is not required to have them.

The Process

1. chef-client asks the server for its solved cookbook list (a list of cookbooks and versions). This may or may not include the core cookbooks, depending on what tools were used to upload the cookbooks.
2. The core cookbooks are prepended to the solved cookbook list, using the solved version if available, or the omnibus-installed version if not.
3. chef-client synchronizes the cookbook cache, downloading anything it doesn't have from the solved cookbook list. NOTE: normally, chef-client would error out if the server does not have a cookbook; but with preinstalled cookbooks, it will ignore this error, since the whole cookbook is already installed.
4. The solved cookbook list is iterated in order (as before) and libraries, resources, providers, attributes, etc. are loaded from them.

Example 1: Simple Runlist

Consider this scenario:

• Node has a run list of base,my_app.
• The my_app cookbook depends on httpd.
• The server has base-1.0.0, httpd-1.5.0 and my_app-1.0.1 (and no other cookbooks).
• chef-client on the node has chef-core-12.6.0, cheffish-1.5.0 and chef-provisioning-2.0.0 preinstalled.

1. chef-client asks the server for its solved cookbook list. It receives base-1.0.0, httpd-1.5.0, my\_app-1.0.1.
2. chef-client adds chef-core-12.6.0, cheffish-1.5.0 and chef-provisioning-2.0.0 to the solved cookbook list.
3. chef-client downloads base-1.0.0, httpd-1.5.0 and my\_app-1.0.1, but not the other three.
4. chef-client loads libraries in this order: chef-core, cheffish, chef-provisioning, base, httpd, my_app.

Example 2: Runlist With Version Pegs

Consider this scenario:

• Node has a run list of base,my_app
• The my_app cookbook depends on cheffish-9.9.9 and httpd.
• The server has base-1.0.0, httpd-1.5.0, cheffish-9.9.9 and my_app-1.0.1 (and no other cookbooks).
• chef-client on the node has chef-core-12.6.0, cheffish-1.5.0 and chef-provisioning-2.0.0 preinstalled.

1. chef-client asks the server for its solved cookbook list. It receives base-1.0.0, cheffish-9.9.9, httpd-1.5.0, my\_app-1.0.1.
2. chef-client prepends chef-core-12.6.0 and chef-provisioning-2.0.0 to the solved cookbook list, and adds cheffish-9.9.9 between them (since the autoloaded cookbooks must load in order, regardless of what version they have).
3. chef-client downloads base-1.0.0, httpd-1.5.0 and my\_app-1.0.1, but not the other three.
4. chef-client loads libraries in this order: chef-core, cheffish, chef-provisioning, base, httpd, my_app.

Policyfile and Berkshelf

Policyfile and Berkshelf will not change. All existing policies and berksfiles will end up using preinstalled core cookbooks (since they do not reference the core cookbooks explicitly).

If a core cookbook is referenced explicitly, both these tools will download appropriate versions from Supermarket, pin them to the run list, and upload them to the server. Since the core cookbooks' versions would be pinned in these cases, the chef-client will receive that version in its solved cookbook list, and download and use it.

Test Kitchen

Test Kitchen will not change. If one of the core cookbooks is specified as a dependency, then it will be explicitly uploaded to Test Kitchen; otherwise, the preinstalled cookbooks will be used.

Copyright

This work is in the public domain. In jurisdictions that do not allow for this, this work is available under CC0. To the extent possible under law, the person who associated CC0 with this work has waived all copyright and related or neighboring rights to this work.