brunoerg

Introduction

Mutation testing is software testing technique that introduces bugs (mutations) on the source code and verify if the automated tests are able to catch these bugs. I created mutation-core (a mutation testing tool) to perform mutation testing on Bitcoin Core codebase. It does not spend time by creating invalid mutants and can deal with some nuances of Bitcoin Core.

However, mutation testing is expensive since we have to compile and run the tests for every single mutant. That is, running mutation testing for every PR may be extremely costly. Instead, we could perform mutation testing once a week based on master branch and make it available for developers to check and decide to write better and/or more tests. That said, I have been doing it and publishing the reports at https://bitcoincore.space.

At this moment, I perform mutation testing for the following critical/more important code: src/wallet/coinselection.cpp, src/script/interpreter.cpp, src/consensus/tx_verify.cpp, `src/consensus/

brunoerg

Mutation testing - #29532

• 64 mutants were generated for the Branch and Bound function.
• coinselector_tests mutation score: 84.37%
• coinselection_tests mutation score: 84.37%

Mutants not killed by coinselection_tests:

diff --git a/src/wallet/coinselection.cpp b/muts/coinselection.mutant.14.cpp

brunoerg

{
  "ipv4": {
    "new": 23959,
    "tried": 47,
    "total": 24006
  },
  "ipv6": {
    "new": 5325,
    "tried": 5,
    "total": 5330

brunoerg

Not killed mutants

5

diff --git a/src/addrman.cpp b/src/addrman.cpp
index f5ca9a5c3..d033297b1 100644
--- a/src/addrman.cpp
+++ b/src/addrman.cpp
@@ -264,7 +264,7 @@ void AddrManImpl::Unserialize(Stream& s_)
     s >> nKey;

brunoerg

#!/usr/bin/env python3
# Copyright (c) 2017-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""An example functional test

The module-level docstring should include a high-level description of
what the test is doing. It's the first thing people see when they open
the file and should give the reader information about *what* the test
is testing and *how* it's being tested