NOTE TO CONTRIBUTORS : Take this wherever you like. If you have articles that really helped you, add them to the list of resources at the end. All content added to this EtherPad is considered public domain. Add your name to the list of contributors at the end of you want attribution.
Suggestions on where to eventually take this content are welcome. I was thinking of making it into a Github pages branch, or maybe on a wiki somewhere?
There seem to be many approaches to test driving your Chef infrastructure, and the amount of projects to pick from is overwhelming. This document tries to provide some overview.
You will need some framework (language, DSL) to write your tests in, this document will call these "test frameworks". These are not specific to Chef. Examples include Minitest, Bats, Cucumber, RSpec.
The tests you write can work on one of two levels. Either they run on the provisioned machine, and make assertions about the actual files, services, etc. (e.g. is there an executable named httpd at /usr/bin). Or they could work on the Chef metadata (e.g. does the package "apache2" have the status :installed). Libraries that provide easy DSLs to make these kind of assertions we will call "test framework extensions". They will be tied to a specific test framework.
To reproducibly run your tests you need transient machines, VMs or cloud servers spun up from a plain base image. Then you provision the machine with your cookbooks, and then run your tests on them.
This is what test-kitchen does, it uses Vagrant to spin up the machine, which can create both local and cloud VMs. It then performs the Chef run, copies the tests, and runs them on the target machine. A different approach is to use a cookbook that acts as a test runner, running the tests within the Chef run.
- bats (shell)
- shunit2 (shell)
- minitest (ruby)
- cucumber (ruby+features)
- rspec (ruby)
- "chef + cucumber = ♥ "
- 22 stars, 10 forks
- https://github.com/iafonov/simple_cuke
- "Reusable Cucumber steps and API for post-convergence system integration descriptions"
- 45 stars, 8 forks
- https://github.com/hedgehog/cuken
- "Continuous Integration testing of ironfan clusters and chef cookbooks. Pass your system diagram into iron law, "
- 16 stars, 6 forks
- https://github.com/infochimps-labs/ironfan-ci
- "Run minitest suites after your Chef recipes to check the status of your system. "
- 111 stars, 33 forks
- https://github.com/calavera/minitest-chef-handler
- Write RSpec examples for Opscode Chef cookbooks
- 196 stars, 60 forks
- https://github.com/acrmp/chefspec
- "RSpec examples and matchers to test your Chef recipes"
- 36 stars, 5 forks
- https://github.com/calavera/rspec-chef
- "RSpec tests for your servers provisioned by Puppet, Chef or anything else even by hand"
- 177 stars, 33 forks
- https://github.com/mizzy/serverspec
- "Framework for test-driven infrastructure development"
- 239 stars, 60 forks
- https://github.com/Atalanta/cucumber-chef
- "This cookbook utilizes the minitest-chef-handler project to facilitate cookbook testing. By default, minitest-handler will collect all the tests in your cookbook_path and run them."
- 41 stars, 30 forks
- https://github.com/btm/minitest-handler-cookbook
- "Framework for running integration tests in an isolated environment "
- 222 stars, 76 forks
Opscode own test system. Uses Vagrant under the hood, and can test against multiple base images (distros) in parallel. Supports all the testing frameworks that Busser supports.
"Kitchen Busser - Runs tests for projects in test-kitchen"
This is a dependency of test-kitchen, it is the part that executes the tests once they've been transferred to the provisioned machine. It has a plugin architecture for supporting various frameworks. So far there is busser-minitest, busser-bats and busser-bash.
Easy VMs for developers. Originally based on VirtualBox but in recent versions provider-agnostic, so can also spin up EC2 instances for instance. Make sure you have the latest version from their site, installs through Rubygems are no longer supported.
When you look at Opscode's cookbooks, a lot of them have a test/cookbooks and test/features directory. The first contains complete Chef cookbooks that contain test files (typically minitest) under the files/ directory. These are copied verbatim to the target machine. The cookbook can do some more setup as preparation for the tests, like creating a test database.
By putting the minitest-chef-handler cookbook in the runlist the minitest files will then get executed at the end of the chef run. A number of helpers are available in modules like Minitest::Chef::Assertions. These test Chef metadata, rather than the underlying system.
The test/features directory contains Cucumber features.
The to be released test-kitchen 1.0 uses a different approach. Rather than running tests inside the Chef run, it first fully provisions the machine, then installs Busser which in turn takes care of running the tests. These styles of tests MUST be in test/integration/ for Busser to find them. The naming further depends on the busser plugin (testing framework) you are using. These are some valid names
- test/integration/default/bats/my_test.bats
- test/integration/foo_suite/minitest/foo_spec.rb
- test/integration/my_recipe/minitest/test_foo.rb
(some ideas for future sections)
- https://github.com/opscode/test-kitchen/wiki/Getting-Started
- http://starkandwayne.com/articles/2013/05/07/tdd-your-devops-with-test-kitchen/
- Arne Brasseur (twitter: plexus) (github: arnebrasseur)