A counter-argument to “Mutation testing with Mutant” using test coverage metrics

.gitignore

*.gem
*.rbc
.bundle
.config
.yardoc
Gemfile.lock
coverage
doc/
tmp

__readme.md

Piotr Solnica argues in “Mutation testing with Mutant” that traditional test coverage is an unsuffiecient metric to consider code being thoroughly tested, and advocates a “mutator” approach (Mutant, Heckle) to discover variance in test cases.

The idea is correct, but the argument and the code in the article is weak. In fact, for the example at hand, code coverage (eg. with SimpleCov) actually does a very good job of communicating weak spots of the code.

In the end, the code either has a purpose, or not. If it doesn't “do anything“, as was the case with the original code in the article, it has not to be properly tested. It has to be removed.

Run this example

$ bundle install
$ bundle exec ruby -I . book_test.rb

book.rb

class Page < Struct.new(:number, :content)
end

class Book
  attr_reader :pages, :index

  def initialize(pages=[])
    @pages = pages
    @index = pages.each_with_object({}) { |page, index| index[page.number] = page }
  end

  def page(number)
    @index.fetch(number) {
      raise "Book does not have a page with number: #{number}"
    }
  end

  def add_page(page)
    @pages << page
    @index[page.number] = page
    self
  end
end

book_test.rb

require 'simplecov' and SimpleCov.start { add_filter "/test|test_/" }

require 'test/unit'
require 'shoulda-context'

require 'book'

class BookTest < Test::Unit::TestCase
  context "Book" do
    subject { Book.new }
    setup   { @page = Page.new(1, 'Test') }

    should "return self" do
      assert_instance_of Book, subject.add_page(@page)
    end

    should "add page to the book" do
      subject.add_page(@page)
      assert_contains subject.pages, @page
    end

    should "add page to the book index" do
      subject.add_page(@page)
      assert_equal @page, subject.index[1]
    end

    should "return the page by number" do
      subject.add_page(@page)
      assert_equal @page, subject.page(1)
    end

    should "raise an error when getting non-existing page" do
      subject.add_page(@page)
      assert_raise(RuntimeError) { subject.page(2) }
    end

  end
end

Gemfile

source 'http://rubygems.org'

gem 'shoulda-context'
gem 'simplecov'