Remove boost as a unit test runner #35587

tl;dr: Make an equivalent test runner to boost with simpler macros and tighter coupling to bitcoin-specific code.

Motivation

There are a number of problems with using dependencies in general. Bugs must either be promptly upstreamed or patched, projects may be abandoned, and they are not tailored to a particular use case. This PR removes boost for writing and running unit tests. In the case of Boost.Test, there are a number of issues which are listed below.

My problems with Boost.Test

Macros

When using boost, there are 3 ways of writing the same thing, each with different outcomes

• BOOST_CHECK(expr1 == expr2): failures show the expression, but no values for expr1 and expr2
• BOOST_CHECK_EQUAL(expr1, expr2): failures show the expression and values, but requires developers remember CHECK_EQUAL and does not use usual operands
• BOOST_TEST(expr1 == expr2): failures show the expression and values, and developers may use familiar operands (==, !=)

All three are preserved to maintain backwards compatibility - presumably, but this results in all 3 being used throughout the repository. As a bonus there is a 4th way, with added context

• BOOST_CHECK_MESSAGE(expr1 == expr2, "the context"): failures show the expression and message, but no values for expr1 and expr2

To make debugging and review easier, all of these should be unified to a single macro. Test sites use operands, print helpful messages - values are always included, and do not vary in approach (deciding between BOOST_TEST, CHECK, and BOOST_CHECK_EQUAL is unintuitive)

<details> <summary>Example boost outputs</summary>

BOOST_AUTO_TEST_CASE(mustfail)
{
    auto a{1};
    auto b{2};
    BOOST_CHECK(a == b);
    BOOST_CHECK_EQUAL(a, b);
    BOOST_TEST(a == b);
    COutPoint outpoint_1{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 0};
    COutPoint outpoint_2{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 1};
    BOOST_CHECK(outpoint_1 == outpoint_2);
    // Doesn't even compile
    // BOOST_TEST(outpoint_1 == outpoint_2);
    // BOOST_CHECK_EQUAL(outpoint_1, outpoint_2);
}

Running 1 test case...
test/argsman_tests.cpp(31): error: in "argsman_tests/mustfail": check a == b has failed
test/argsman_tests.cpp(32): error: in "argsman_tests/mustfail": check a == b has failed [1 != 2]
test/argsman_tests.cpp(33): error: in "argsman_tests/mustfail": check a == b has failed [1 != 2]
test/argsman_tests.cpp(36): error: in "argsman_tests/mustfail": check outpoint_1 == outpoint_2 has failed

</details>

<details> <summary>New outputs</summary>

TEST_CASE(mustfail)
{
    auto a{1};
    auto b{2};
    CHECK(a == b);
    COutPoint outpoint_1{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 0};
    COutPoint outpoint_2{Txid{"0000000000000000000000000000000000000000000000000000000000000100"}, 1};
    CHECK(outpoint_1 == outpoint_2);
}

Running 1 test cases...
[FAIL]: test/argsman_tests.cpp:30: CHECK(a == b)
1 == 2

[FAIL]: test/argsman_tests.cpp:33: CHECK(outpoint_1 == outpoint_2)
COutPoint(0000000000, 0) == COutPoint(0000000000, 1)

[FAIL] mustfail (2/2 checks failed)

</details>

No repository context

Many types implement ToString, but boost has no way of meaningfully using these representations (will change w/ future std::format). With a repository-specific test runner we can use these today. See example above.

Not extensible

With the test runner in this repository we can add things such as #35139, implement string representations for more types, and implement ideas from #8670.

Other issues

More issues with boost are raised in #34666, #8670. This PR does not resolve all of them, but one it does address immediately is banning comparisons of integers with different signs, previously allowed by boost. Using a test runner within the source also makes IWYU easier to work with.

High level changes

For those running tests, not a tremendous amount has changed in this PR. For instance, the doc page remains valid. Changes to the runner options include anything that is not help, run_test, log_level, list_content. This includes catch_system_errors which is always no. The log_levels have been reduced to 5 levels.

Changes in writing tests

• CHECK to compare two values with any ==, !=, >, etc
• CHECK(a == b, "my message") to append a message
• REQUIRE, same as CHECK, but will fail the test immediately
• CHECK_EQUAL_RANGES(it1, it2) to compare two ranges
• TEST_CASE(name) to add a test
• FIXTURE_TEST_CASE(name, Fixture) to add a test with a fixture
• TEST_SUITE_BEGIN/END to declare a suite, optionally with fixture for each test in the suite

Migration

Boost.Test is removed in this PR, but the macros are aliased by BOOST_* counterparts so there are no merge conflicts. The idea would be to migrate test files to the new macros when there are no/low number of conflicts on that file.

<!--e57a25ab6845829454e8d69fc972939a-->

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

<!--006a51241073e994b41acfe9ec718e94-->

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35587.

<!--021abf342d371248e50ceaed478a90ca-->

Reviews

See the guideline and AI policy for information on the review process.

If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

<!--174a7506f384e20aa4161008e828411d-->

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #35569 (Encapsulation for CTransaction by purpleKarrot)
• #35531 (txindex: hash keys and pack positions to reduce disk usage by andrewtoth)
• #35195 (coins: cache UTXO outpoint hash codes by l0rinc)
• #34872 (wallet: fix mixed-input transaction accounting in history RPCs by w0xlt)
• #34864 (coins: tighten cache entry state invariants by l0rinc)
• #34075 (fees: Introduce Mempool Based Fee Estimation to reduce overestimation by ismaelsadeeq)
• #33540 (argsman, cli: GNU-style command-line option parsing (allows options after non-option arguments) by pablomartin4btc)
• #31682 ([IBD] specialize CheckBlock's input & coinbase checks by l0rinc)
• #29278 (Wallet: Add maxfeerate wallet startup option by ismaelsadeeq)
• #17783 (common: Disallow calling IsArgSet() on ALLOW_LIST options by ryanofsky)
• #17581 (refactor: Remove settings merge reverse precedence code by ryanofsky)
• #17580 (refactor: Add ALLOW_LIST flags and enforce usage in CheckArgFlags by ryanofsky)
• #17493 (util: Forbid ambiguous multiple assignments in config file by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

<!--5faf32d7da4f0f540f40219e4f7537a3-->

LLM Linter (✨ experimental)

Possible typos and grammar issues:

• evalution -> evaluation [misspelled in the comment “captures the expression before evalution”]

Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

• GetSetting(settings, "section", "name", false, false, false) in src/test/settings_tests.cpp
• GetSetting(settings, "section", "name", false, false, false) in src/test/settings_tests.cpp
• Test(ms_stack_limit, "?", "?", TESTMODE_VALID | TESTMODE_NONMAL | TESTMODE_NEEDSIG | TESTMODE_P2WSH_INVALID, 4 * count + 1, 1, {}, {}, 1 + count + 1) in src/test/miniscript_tests.cpp

^{2026-07-08 17:54:02}

Strong Concept ACK

My problems with boost

Boost may have multiple problems on its own, but the problems that you list are specific to Boost.Test. You should not conflate the two.

CHECK_EQUAL_RANGES(it1, it2) to compare two iterators

Iterators or Ranges?

TEST_SUITE_BEGIN/END to declare a suite

Boost.Test allows arbitrary deep nesting of test suites. When integrating Boost.Test into CTest, this causes headaches for little benefit. I hope we can follow-up on this and restrict the test hierarchy to exactly one test suite (no more and no less) using something along the TEST(TestSuiteName, TestName) macro from GoogleTest.

Why no concurrency?

In my opinion, concurrency does not belong in a testing framework. Many testing frameworks have a test driver built-in. But separating test framework and test driver into two orthogonal components allows much more powerful composition. I would even go so far and claim that a test framework should not provide a way to run more than one test; neither concurrently, nor sequentially.

What a test framework needs to be able to provide, is a way to launch a single test and a way to tell the test driver about the list of tests that are available. The test driver can then orchestrate tests from different testing frameworks from potentially different programming languages and invoke tests individually, sequentially, concurrently, or even distributed across multiple hosts.

This kind of integration is exactly the reason why I implemented discover_tests, which will be available in CMake 4.4.

The Boost.Test framework is notoriously bad at this. When selecting a single test, it spams out information about all tests that are not selected. While it can provide information about what tests are available, that output is optimized for human readability and very hard to parse by a test driver. We can definitely do better with a custom test framework!

There are a number of problems with using dependencies in general. Bugs must either be promptly upstreamed or patched, projects may be abandoned, and they are not tailored to a particular use case. bitcoind has a small number of dependencies and will have fewer very soon (https://github.com/bitcoin/bitcoin/pull/34411). One of the final remaining dependencies is boost, which is responsible for the unit tests and multi-index. This PR removes boost for writing and running unit tests.

Boost.Test is not a dependency of bitcoind, only of test_bitcoin so this is not comparable to the libevent removal. Also boost::multi_index removal was rejected very recently: #34586.

only of test_bitcoin

Correct me if I'm wrong but it appears the config here still builds test_bitcoin: https://github.com/bitcoin/bitcoin/blob/master/contrib/guix/libexec/build.sh#L125

Also boost::multi_index removal was rejected very recently: #34586

That approach used STL containers. An actual replacement would need to use something like this: https://www.mail-archive.com/digitalmars-d@puremagic.com/msg33920.html

Correct me if I'm wrong but it appears the config here still builds test_bitcoin

Yes, but what does that have to with my comment? I just said that it's a dependency of test_bitcoin not bitcoind. Reducing the dependencies of bitcoind is a lot more interesting than those of test_bitcoin for obvious security reasons. I don't think the fact that test_bitcoin is built in a release and installed changes much of that. Maybe it does for other people but I think your motivation should be more clear on this if this is what you are after. Particularly this part:

bitcoind has a small number of dependencies and will have fewer very soon (https://github.com/bitcoin/bitcoin/pull/34411). One of the final remaining dependencies is boost, which is responsible for the unit tests and multi-index.

I would read this very clearly as boost (including Boost.Test) is a dependency of bitcoind if I didn't know better.

That approach used STL containers.

The pull was rejected on conceptual basis, not approach, so I don't think this matters.

I would read this very clearly as boost (including Boost.Test) is a dependency of bitcoind if I didn't know better.

Sure, PR description updated.

Reducing the dependencies of bitcoind is a lot more interesting than those of test_bitcoin for obvious security reasons

I find this less obvious then the comment implies. A repository going stale is far more likely than some backdoor or high severity CVE, so "a lot more interesting" feels subjective.

Yes, but what does that have to with my comment?

A lot. @rustaceanrob correctly points out that even though Boost.Test is not strictly a dependency of bitcoind, it is included in the release flow that creates and packages bitcoind, along with test_bitcoin. It is correct to say that Boost.Test is a dependency of our release process.

A repository going stale is far more likely than some backdoor or high severity CVE, so "a lot more interesting" feels subjective.

Precisely. It's also worth remembering the xzutils backdoor was introduced through test code.

Strong Concept ACK!

Love to see it. While I broadly agree with the author's reasons stated in the PR, the ones I'll highlight specifically are:

• No repository context
• Not extensible

Having our own simple, in house test framework gives us the ability to customise it for exactly our needs. I believe we should do everything in our power to make testing an easy, intuitive, and feature rich experience as this helps to remove barriers to people writing and interpreting tests.

In particular, as a longer term project I am looking at ways to improve concurrency in our testing to help speed up CI. As @purpleKarrot points out:

The Boost.Test framework is notoriously bad at this. When selecting a single test, it spams out information about all tests that are not selected. While it can provide information about what tests are available, that output is optimized for human readability and very hard to parse by a test driver. We can definitely do better with a custom test framework!

996832947dd1053b2e31acd1ef3e2933804fd580 "test: Separate mock_process into a standalone binary"

This approach seems incompatible with the release process.

For example, I downloaded the bitcoin-e6e8d1f00a08-win64-unsigned.zip "release" binaries on a Windows machine, unpacked them, and ran:

>.\bitcoin-e6e8d1f00a08\libexec\test_bitcoin.exe
Running 770 test cases...
EXCEPTION in run_command: CreateProcess failed: The system cannot find the file specified.
[FAIL] run_command (0/1 checks failed)
[WARN]: Variable DIR_UNIT_TEST_DATA unset, skipping script_assets_test

770 tests: 769 passed, 1 failed, 0 skipped (26566129 checks)

This approach seems incompatible with the release process.

a0d535a78b63df4d04411efb1b2c0f98ee87211e uses an environment variable to determine if the test should run as the mock process or the actual test runner. No added files and should not effect the release.

Concept ACK 7974316a1a5e9e84be446f86fed52564ee954b4b

This change has enough pro's for me to support it;

• Unification of all the different BOOS_CHECK*
• Better orchestrate-able, see #35587 (comment)
• Removal of dependency.

The pull was rejected on conceptual basis, not approach, so I don't think this matters. @fjahr this is not an accurate framing of the discussion on #34586. It is acknowledged that fewer dependencies are desirable, if the replacements are an improvement. The critique that follows is specifically about the use of STL containers (which are indeed worse). The second push back from a different reviewer echos the sentiment: boost multi index is better than the alternatives (STL containers, in this case).

I think this PR stands on its own merit as better than using Boost.Test, regardless of whether or not we end up fully removing boost. However, if a better alternative to boost multi index were to be proposed (e.g., @rustaceanrob 's link), this PR and that perhaps future PR only strengthen each other, not weaken.

@rustaceanrob if you want, couple commits here to prune out the Boost Test dep: https://github.com/fanquake/bitcoin/commits/35587_drop_pkg/.

Nice, I picked those up, thanks

Concept ACK

I like the changes, but I wonder if another approach could be done here, because replacing the check macros and replacing the framework are not tied together and can be done subsequently and independently.

Given that this has ~100 direct merge conflicts and possibly a few more silent merge conflicts, I wonder if this should be made easier to review and rebase against.

My preference would be to:

• First add the new macros (and the required changes to use them), similar to #35139
• Then, switch to them in a commit that is easier to review than a commit that also changes the framework in the same diff.
• Then introduce the new framework.
• Then switch to the new framework and remove boost.

Just my preference, though. Not sure what other reviewers prefer.

Edit: An alternative approach could be to keep this pull as-is, but drop the large "macro rewrite" and instead provide legacy aliases, which can be used temporarily (#define BOOST_CHECK_EQUAL(a, b) CHECK(a == b) // or CHECK_EQUAL(a, b) or whatever the name of the new macro is). This way, many conflicts/rebases during the review of this pull can be avoided, which should make re-review less tedious here. The "simple"/mechanical replacement can then be done later, hopefully at a pre-defined time to minimize the conflicts/rebases and thus minimize the re-review load again?

An alternative approach could be to keep this pull as-is, but drop the large "macro rewrite" and instead provide legacy aliases, which can be used temporarily (#define BOOST_CHECK_EQUAL(a, b) CHECK(a == b) // or CHECK_EQUAL(a, b) or whatever the name of the new macro is).

Makes sense

Updated with legacy aliases

This only has 24 comments, but is already tedious to review on GitHub, because GitHub decides to hide even a 4 day old comment (https://github.com/bitcoin/bitcoin/pull/35587#discussion_r3486342364). This GitHub behavior was triggered by the (no longer relevant) long list of conflicts.

Maybe open as a fresh pull request? This should be easy, given that there are no outstanding review comments. (Except maybe mine)

<img width="1010" height="78" alt="Screenshot 2026-07-01 at 10-30-44 Remove boost as a unit test runner by rustaceanrob · Pull Request [#35587](/github-metadata-backup-bitcoin-bitcoin/35587/) · bitcoin_bitcoin" src="https://github.com/user-attachments/assets/b54ef02f-863f-4a42-9a38-17a513f36f10" />

This only has 24 comments, but is already tedious to review on GitHub

Maybe open as a fresh pull request?

We need to find a more sustainable/scalable way to do this; GitHub is basically unusable for non-trivial PRs.

GitHub decides to hide even a 4 day old comment

https://github.com/refined-github/refined-github has a browser extension to work around that (command+clicking these opens all hidden sections). I also added automatic open to my helper in https://github.com/l0rinc/ACKtopus.

This only has 24 comments, but is already tedious to review on GitHub Maybe open as a fresh pull request?

We need to find a more sustainable/scalable way to do this; GitHub is basically unusable for non-trivial PRs.

I think here it is really only related to the long list of conflicting pulls. I guess each conflict was an "event" (like a comment). E.g., #35402 is another pull request with roughly 24 comments, and it works just fine.

I think here opening a fresh pull is trivial and sufficient, but of course I won't object moving off of GitHub :sweat_smile:

GitHub decides to hide even a 4 day old comment

https://github.com/refined-github/refined-github has a browser extension to work around that (command+clicking these opens all hidden sections). I also added automatic open to my helper in https://github.com/l0rinc/ACKtopus.

Thx. Browser extensions, in combination with javascript and LLM integration give me a bit of unease. I think I'll just click the mouse button thrice for now here :sweat_smile:

A third alternative could be to not trigger GitHub events for each conflict, by using a click-jacking URL. Opened a brainstorming issue in https://github.com/maflcko/DrahtBot/issues/83

Concept ACK

A net benefit, both due to reducing Boost usage and improvements (current and future) to the unit test framework.

Some minor/quick observations:

• It's a large diff, as all test code is affected. Could be broken up as: introduce new framework, change testers (in chunks), remove boost framework leftovers. Testers that are touched in pending PRs could be left at the end.
• Once this is over, the macros could be renamed to not have the BOOST prefix (scripted diff)
• I've seen some seemingly unrelated change to non-test code (const -> constexpr)

Not yet sure about this, another huge change (that kills the GitHub UI) - I'd be interested in a gradual/minimal approach here that doesn't touch so many files.

I think it is nice to have a self-contained and complete patchset that applies globally all at once. Of course, if there are specific unrelated changes that truly make sense on their own and where it makes sense to split them up, it can be done. However, some of the changes seem not needed and possibly even going in the wrong direction, as pointed out in review (https://github.com/bitcoin/bitcoin/pull/35587#discussion_r3486342364, or #35587 (review), or #35625 (comment)). So there is somewhat of a chance that some of those may be dropped and the result here will be slim enough to go in as mostly a single pull request.

<!--85328a0da195eb286784d51f73fa0af9-->

🚧 At least one of the CI tasks failed. _{Task previous releases: https://github.com/bitcoin/bitcoin/actions/runs/28954672394/job/85910327486} _{LLM reason (✨ experimental): CI failed due to a C++ build error: a static_assert in /usr/include/c++/12/utility (from std::cmp_equal) when compiling src/test/util_tests.cpp.}

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

• Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

• A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

• An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>