Idea: combine the functional tests with fuzzing.
1. Existing functional tests carry bias:
E.g.: a single line modification to one of the compact block relay tests triggers CVE-2024-35202.
diff --git a/test/functional/p2p_compactblocks.py b/test/functional/p2p_compactblocks.py
index ca36b2fbc0..cd18951435 100755
--- a/test/functional/p2p_compactblocks.py
+++ b/test/functional/p2p_compactblocks.py
@@ -553,6 +553,7 @@ class CompactBlocksTest(BitcoinTestFramework):
msg = msg_blocktxn()
msg.block_transactions = BlockTransactions(block.sha256, [block.vtx[5]] + block.vtx[7:])
test_node.send_and_ping(msg)
+ test_node.send_and_ping(msg)
# Tip should not have updated
assert_equal(int(node.getbestblockhash(), 16), block.hashPrevBlock)Fuzz tests are good at finding bugs because they reduce bias.
2. Reducing review strain on refactors:
When introducing new features, improving performance or writing unit/fuzz tests we often end up refactoring the relevant code first, as the code base lacks modularity or because existing abstractions weren't quite right. For example:
- BIP324 involved refactoring the networking transport abstraction: bitcoin/bitcoin#28165
- TxDownloadMan encapsulates logic from net processing into its own module: bitcoin/bitcoin#30110
- Changing LevelDB file size: bitcoin/bitcoin#30039
- ...
The risk of introducing bugs during these refactors is higher and results in more manual review overhead if:
- there are no tests in place already that characterize the behavior of the code being changed
- existing tests need to be modified, due to e.g. interface changes
Our functional tests operate at the outermost level possible for tests. They spin up actual bitcoinds and test through the external interfaces (P2P, RPC, etc). Refactoring is almost never required to write a functional test, at most new RPCs are added for state inspection. To some degree they therefore provide protections against bugs introduced through refactoring but they carry bias (see 1.) and I believe we can significantly expand on their protections by combing the functional tests' approach with fuzzing.
The rough architecture of this approach looks as follows:
The virtual machine used here has the special ability to create a snapshot of itself (CPU registers, memory, devices (disk, ...)) and to quickly reset itself to this snapshot at any time. This allows us to:
- setup any kind of initial state we need for a test without incurring the performance penalty of having to do the setup every fuzz iteration (e.g. setup a network of nodes, mine an initial chain, populate the mempool, ...)
- avoid state to build up in the bitcoind instances under test across fuzz iterations -> determinism
This concept is called snapshot fuzzing and the prototype uses Nyx for this at the moment (other tools exist as well).
The "Harness" is the test that executes fuzzer supplied test cases against the target under test.
The approach outlined here allows to create some advanced testing scenarios.
Bitcoin Core's behavior can be characterized through differential fuzzing of different versions of the code (e.g. master vs. PR branches, other impls), which can detect unintended changes in behavior, such as: consensus bugs, p2p changes, rpc breakage, etc (subject to test coverage). This type of test specifically could reduce the risk described in "Motivation 2".
Given two connected nodes, it should not be possible for an attacker to connect to one of them and force a disconnection of the other node. Using a functional fuzz test we can test this invariant:
Prototype?
Here: dergoegge/fuzzamoto (private repo for now)
Does this obsolete refactoring and writing isolated fuzz tests?
No, but it helps reduce the risk of those changes!
Coverage guidance still possible?
Yes! Both compile time instrumentation and full system coverage collection (using intel-pt) is supported by Nyx.
Why not use Python and reuse the existing framework?
The prototype is written in Rust but it shouldn't be too hard to get this working with the existing Python test framework as well. The python framework might be a better choice in terms of mindshare across the contributor base.
The primary reason for not using Python is that I prefer Rust. Plenty of Rust Bitcoin crates exist that make matching the python framework not too hard.
Is the goal to have this merged into Bitcoin Core?
Maybe in the future if it gathers similar traction to the existing functional tests but it is not part of the goals at the moment.
Does something similar exist for other projects?
Sort of: Fuzzilli for JavaScript engine fuzzing & Syszkaller for linux kernel fuzzing.