README for Muzak

README.md

Muzak

What is Muzak?

Muzak is a mutation testing library for Elixir and Erlang applications.

What is Mutation Testing?

Mutation testing is a way of systematically introducing bugs into your code and then running your tests to see if any of them fail. If you introduce a bug in your code and none of your tests fail, then either

1. You're missing a test, or
2. You have dead code that you can delete

Using Muzak

To use Muzak you don't need to make any changes to your code other than adding the library to your dependencies in your mix.exs file like so:

defp deps() do
  [
    # ...
    {:muzak, "~> 1.0", organization: "muzak"}
  ]
end

Once you've done that and fetched your dependencies, you can now run mix muzak to run your mutation tests - and that's it! There is some optional configuration that can be set for Muzak Pro and Muzak Enterprise, but getting started is really that easy.

Features

There are three different versions of Muzak, each with a different feature set.

Muzak

• A basic set of mutators
• Configuration to set a maximum number of mutations to apply per line of code
• Configuration to include or excude functions, modules, files or directories from being mutated
• Ability to run only a single test file or a single test against your mutations
• Determinisitc, repeatable results
• Clear, helpful CLI failure messages
• Extensive documentation

Muzak Pro

• All the features of Muzak
• An additional, extensive set of mutators
• Ability to configure the behavior of certain mutators
• Configuration to allow mutations to only be applied to lines that have been changed in the current commit or since the previous merge commit
• High quality parallelism support to reduce runtime
• Advanced selection of which code to mutate to reduce runtime
• Easy-to-use HTML reports
• Direct email support for questions or issues

Muzak Enterprise

• All the features of Muzak Pro
• Ability to define your own application-specific mutators
• Advanced selection and application of mutators to find the mutations most likely to quickly produce a failure
• Ability to ignore certain known, acceptable failures for future test runs
• A 1 hour concierge onboarding session (over video call) with your team to teach them how to best use Muzak to suit their needs
• Priority support for questions or bug reports over email

An example

Imagine we have the following module:

defmodule Calc do
  def maybe_add(num) do
    if num in [2, 3, 5, 8, 11, 19] do
      num
    else
      num + 1
    end
  end
end

and the following tests for that module:

defmodule CalcTest do
  test "does not add 1 to some fibonacci numbers" do
    assert Calc.maybe_add(3) == 3
  end

  test "adds 1 to non-fibonacci numbers" do
    assert Calc.maybe_add(4) == 5
  end
end

With those two tests we've achieved 100% code coverage of that function, but we can see there are still many ways to break that code without producing a failed test. Muzak will illustrate this for us by making a single change in the code and then running all the tests again. If none of the tests fail for a given change, that's an indication that we have another test we can add.

Some examples of mutations that muzak will make are:

defmodule Clac do # <= rename atom
  def maybe_add(num) do
    if num in [2, 3, 5, 8, 11, 19] do
      num
    else
      num + 1
    end
  end
end

defmodule Calc do 
  def daa_ebyam(num) do # <= rename function
    if num in [2, 3, 5, 8, 11, 19] do
      num
    else
      num + 1
    end
  end
end

defmodule Calc do
  def maybe_add(num) do
    if num not in [2, 3, 5, 8, 11, 19] do # => change `in` to `not in`
      num
    else
      num + 1
    end
  end
end

defmodule Calc do
  def maybe_add(num) do
    if num in [2, 3, 5, 8, 11, 19] do
      num
    else
      num - 1 # => change `+` to `-`
    end
  end
end

defmodule Calc do
  def maybe_add(num) do
    if num in [2, 3, 5, 8, 11, 19] do
      num
    else
      num + 0 # => change integer literal
    end
  end
end

defmodule Calc do
  def maybe_add(num) do
    if true do # => replace condition with boolean
      num
    else
      num + 1
    end
  end
end

defmodule Calc do
  def maybe_add(num) do
    if num in [11, 3, 19] do # => Change list literal
      num
    else
      num + 1
    end
  end
end

These are just some of the many ways in which muzak will alter your code before re-running your tests, and with Muzak Enterprise you can even write custom mutators to make application-specific changes.

When we run mix muzak, most of those mutations will result in at least one failed test, and this counts as a successful run, but we can see that the final change illustrated above will not result in a test failure, and so we will see the following failure output:

This should give us some hints about the test cases that we're missing, and so we can now update our tests to cover everything that's currently missing:

defmodule CalcTest do
  test "does not add 1 to some fibonacci numbers" do
    for num <- [2, 3, 5, 8, 11, 19] do
      assert Calc.maybe_add(num) == num
    end
  end

  test "adds 1 to non-fibonacci numbers" do
    assert Calc.maybe_add(4) == 5
  end
end