ci: add [free|open|net]BSD ci test jobs #36147

willcl-ark commented at 11:07 AM on September 2, 2026: member

We currently cross compile various BSD binaries but do not test them. This leads to regressions on master which are noticed in nightly builds and fixed here, recent examples include #36078 and #36145.

Build and test the binaries in this repo as part of our CI instead to avoid this regression/fix flip-flopping.

As we don't have native *BSD runners we require BSD VMs to test the binaries. https://github.com/vmactions offers these (and are currently being used in various nightly jobs).

Unfortunately it's not possible to run all of the cross-compiled binaries directly in these VMs because the VM image builds (via anyvm) are syspatching their images, which cause some function calls to be missing when executing the (pure) cross-compiled binaries.

Therefore here, and to keep all BSD builds following a single CI process, build and test all BSDs inside their VM.

The VM caches itself after its prepare step (installing dependencies) so operates similarly to an unchanged docker buildkit image stage.

A clean (no caches) build and test inside the VM on a free GHA runner takes ~ 60 minutes. With the various caches seeded this drops to about 20-30 minutes. On a larger Warp runner I'd expect both of these runtimes to be much shorter, but this is untested currently.

There is some slowness when the tests are completed: rsync copies various cache directories back out from the VM in case they want to be GHA-cached later. It may be possible to optimise this slowness a little in the future.

The patchset also includes two small fixes, one needed to get the tests running, and one to prevent an overly-long REST request hang observed during testing (it may be possible/preferable to drop the REST request timeout reduction before merging).

It also omits feature_reindex_init and p2p_private_broadcast_retry_v1 from the functional tests; the former fails, and the latter seems to take an extraordinarily long time. Both might be good candidates for investigation and followup PRs.

These jobs will require enabling 3 more actions:

vmactions/freebsd-vm@v1 vmactions/netbsd-vm@v1 vmactions/openbsd-vm@v1

I have tried to re-use or at least mimic the current CI execution model/flow wherever possible.

test: silence lsof warnings on FreeBSD

FreeBSD lsof emits warnings even though the requested socket data remains
usable. Suppress them to keep functional test output focused.

4b890c5b30

test: bound REST request timeout

A stalled REST response can otherwise hang the entire functional test job.
Use a finite socket timeout so the request fails with a diagnostic.

9e72629be6

ci: support native BSD test runs

Reuse the existing host-mode CI driver instead of maintaining a second entry
point. BSD hosts arrive pre-provisioned, while the shared scripts handle the
native tools and resource limits.

6be2f51770

ci: replace BSD cross jobs with native jobs

The BSD VMs update their base systems, so cross-built binaries can target
stale interfaces. Build and test within each VM while retaining the standard
test runner and repository caches.

31a2ac55c0

DrahtBot added the label Tests on Sep 2, 2026

DrahtBot commented at 11:07 AM on September 2, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/36147.

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
Concept ACK	hebasto

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#36100 (ci: use LLVM 23 in *san, fuzz, *cross jobs by fanquake)
#36082 (ci: add Guix builds to CI by willcl-ark)
#36052 (ci: Doc: Move all config comments right next to the option they explain by maflcko)
#36042 (build: Bump g++ minimum supported version to 13 by maflcko)
#35774 (ci: test cross-built macos arm64 binaries by willcl-ark)
#35762 (test: optionally run functional tests via CTest by willcl-ark)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

maflcko commented at 12:11 PM on September 2, 2026: member

We currently cross compile various BSD binaries but do not test them. This leads to regressions on master which are noticed in nightly builds and fixed here, recent examples include #36078 and #36145.

Build and test the binaries in this repo as part of our CI instead to avoid this regression/fix flip-flopping.

I think this requires a stronger motivation, to explain that this will actually reduce overall workload.

Those vm-in-a-vm tests are extremely brittle (https://github.com/hebasto/bitcoin-core-nightly/issues/230#issuecomment-4311704411, https://github.com/hebasto/bitcoin-core-nightly/issues/81, ... etc)

So if those tasks are added here, we need to understand that this is the "hot path" (for all pull requests) and before this is merged, there needs to be several people who monitor those intermittent failures on those platforms and then categorize them into "intermittent", "unrelated", etc.

If this isn't done, then devs will just ignore all CI results because the false-positive rate is too high and just re-run it until it is green. This may then degrade non-*BSD code at a slight benefit of avoid some trivial test-only BSD fixups.

Maybe I am wrong, but I wanted to raise the point to not miss the large picture here.

hebasto commented at 12:24 PM on September 2, 2026: member

We currently cross compile various BSD binaries but do not test them. This leads to regressions on master which are noticed in nightly builds and fixed here, recent examples include #36078 and #36145. Build and test the binaries in this repo as part of our CI instead to avoid this regression/fix flip-flopping.

I think this requires a stronger motivation, to explain that this will actually reduce overall workload.

Those vm-in-a-vm tests are extremely brittle (hebasto/bitcoin-core-nightly#230 (comment), hebasto/bitcoin-core-nightly#81, ... etc)

So if those tasks are added here, we need to understand that this is the "hot path" (for all pull requests) and before this is merged, there needs to be several people who monitor those intermittent failures on those platforms and then categorize them into "intermittent", "unrelated", etc.

If this isn't done, then devs will just ignore all CI results because the false-positive rate is too high and just re-run it until it is green. This may then degrade non-*BSD code at a slight benefit of avoid some trivial test-only BSD fixups.

Maybe I am wrong, but I wanted to raise the point to not miss the large picture here.

We might begin with FreeBSD only, which has shown high stability over a long period.

maflcko commented at 12:38 PM on September 2, 2026: member

We might begin with FreeBSD only, which showed a high stability over a long period.

Your link shows an unrelated failure for the 35th build (second page). Not sure if this is accurate, but if every 35th build fails and Bitcoin Core has 35 pushes per day, then every day there will be one additional unrelated CI failure.

Maybe there can be a one-month trial period and then a summary is presented with all CI failures and each of them annotated with "intermittent" (y/n), "unrelated" (y/n)?

Also, even if a failure is related, it can still be intermittent, so merging this pull request does not guarantee to prevent fixups such as #36145, because the failure happens intermittently.

willcl-ark commented at 2:18 PM on September 2, 2026: member

I did not understand these VMs to be flaky. I didn't observe any flakiness in testing, but probably only ran about 15 runs (x3 jobs), so perhaps I'm due a random flake-failure soon 😋

The actions are used by a few other high-profile projects like tokio and sccache as well as:

tukaani-project/xz using OpenBSD (workflow)
hashcat/hashcat using OpenBSD
acmesh-official/acme.sh using OpenBSD
aws/aws-lc-rs using FreeBSD (workflow)
mozilla/sccache using FreeBSD
aristocratos/btop using OpenBSD
stalwartlabs/stalwart using FreeBSD
deskflow/deskflow using FreeBSD
FrameworkComputer/framework-system using FreeBSD

But I'd imagine our functional tests put significantly more strain on the vm than any of those projects are doing.

I asked codex to take a look at the last months-worth of runs on hebasto's nightly repo:

<details> <summary>Details</summary>

hebasto/bitcoin-core-nightly from 2026-08-02 through 2026-09-02. Pull-request runs were excluded. The nightly sample consists of runs with the workflow_run event. Manual workflow_dispatch failures are listed separately where they add useful context.

Summary

Workflow	Nightly runs	Successful	Failed	Cancelled
FreeBSD	33	33	0	0
NetBSD	33	28	4	1
OpenBSD	33	10	19	4

The logs for all 29 failed runs were downloaded and inspected.

FreeBSD looks stable in this sample. NetBSD has a small number of nightly failures, mostly caused by platform resource limits or platform-sensitive tests. OpenBSD has a high failure rate, but most failures are repeatable test or platform problems rather than failures to boot the VM.

NetBSD

Nightly failures

Run 906, 2026-08-25: feature_block.py timed out waiting for the expected peer disconnect. Other jobs showed the NetBSD file-descriptor limit.
Run 899, 2026-08-20: feature_block.py timed out waiting for a peer disconnect after the node had rejected the block.
Run 892, 2026-08-14: wallet_listtransactions.py failed when generateblock returned bad-txns-inputs-missingorspent.
Run 888, 2026-08-10: wallet_listtransactions.py failed with the same TestBlockValidity error.

The feature_block.py failures look like delayed networking or scheduling in the guest: the node log records the rejection and disconnect, while the test does not observe the disconnect before its timeout. The wallet failures are functional test failures and should not be counted as generic VM boot failures.

Manual failures

The following manual runs on 2026-08-25 failed during test setup because the guest did not have enough file descriptors:

The common messages were:

Not enough file descriptors available. 256 available, 290 required.
Warning: Reducing -maxconnections ... because of system limitations.

These are reproducible NetBSD configuration or test-environment failures, not random VM failures.

There was one additional manual failure:

Run 895, 2026-08-16: the VM boot probe timed out and QEMU was restarted. The run also contained feature_anchors.py and interface_http.py timing failures.

This is the clearest NetBSD example of actual VM or infrastructure flakiness.

OpenBSD

Repeated `interface_gui.py` failures

interface_gui.py failed in runs 738, 739, 740, 743, 744, 745, 746, 747, 748, 749, and 751.

The recurring error was:

GLib-CRITICAL: g_main_context_pop_thread_default:
assertion 'stack != NULL' failed

This is highly repeatable across OpenBSD 7.8 and 7.9 GUI jobs. It looks like an OpenBSD/Qt shutdown or test-environment problem, not QEMU randomly failing.

Repeated P2P test timeouts

p2p_private_broadcast.py or p2p_private_broadcast_retry_v1.py failed in runs 738, 754, 757, 758, 761, 765, 766, 772, and 779.

The test waited exactly 480 seconds for all addresses to advertise NODE_P2P_V2:

Predicate ... not true after 480.0 seconds

The node remained responsive and answered getnodeaddresses RPC requests, but the expected service flags never appeared. Because the same test fails in the same way across many consecutive OpenBSD runs, this looks like an OpenBSD/test interaction or severe guest-performance problem rather than a generic boot failure.

Package and VM infrastructure failures

Run 754: OpenBSD package installation failed with Premature end of archive while installing nghttp2.
Run 761, 765, and 766: package downloads included 404s or connection timeouts.

These failures are package-mirror or network infrastructure failures. They are separate from the repeated functional test failures.

Conclusion

For this one-month sample:

FreeBSD: no observed nightly failures.
NetBSD: some real guest or scheduling sensitivity, but most failures are resource-limit or functional-test issues. The one clear VM boot failure was on 2026-08-16.
OpenBSD: 19 failed nightly runs out of 33. The failure count is dominated by repeatable interface_gui.py and p2p_private_broadcast* failures, with a few package-mirror failures. It would be misleading to attribute all 19 to vmactions or QEMU flakiness.

The next useful experiment would be to fix or disable the known OpenBSD GUI and P2P test incompatibilities, raise the NetBSD file-descriptor limit, and then repeat the same nightly comparison. That would leave a much cleaner measurement of VM-level reliability.

</details>

I have not observed the interface_gui.py failures that hebasto's repo sees on OpenBSD. I did see a failure on p2p_private_broadcast_retry_v1.py once or twice and did have that test disabled on *BSD in one iteration; if that test in particular is flaky it could be disabled on these jobs.

Most of the other failures at first glance appear to be GHA/package archive 404 or other network infra failures and I wouldn't attribute them to the VMs being flaky, so I'm not 100% convinced that critique is warranted.

That said I don't want to add known-flaky jobs and you are correct to call out they could be more work to maintain than porting over a few fixes, if they are unreliable. I suppose the only unclear bit is at what cost level do we prefer to merge buggy code and fix later, vs run more jobs on every PR and merge to prevent that happening in the first place...

We might begin with FreeBSD only, which has shown high stability over a long period.

Happy to start with FreeBSD only if that's preferable.

maflcko commented at 2:47 PM on September 2, 2026: member

We might begin with FreeBSD only, which has shown high stability over a long period.

Happy to start with FreeBSD only if that's preferable.

Sure, I am not against this pull request, but then the motivation is off, because the two motivating issues are about netbsd and openbsd, which for some reason have a lot more false positive intermittent failures?

It still seems that the remaining fixups required for FreeBSD are less effort than maintaining the FreeBSD ci task here? (Recall that codex only looked at the last month with daily builds, but it missed the failure from 35 days ago and recall that Bitcoin Core has a higher CI volume than just one build per day, so any rare failure will amplify)

Again, seems fine to merge this, but as long as the false positive rate is higher than the true positive rate, it doesn't seem worth it overall.

DrahtBot added the label Needs rebase on Sep 3, 2026

DrahtBot commented at 11:54 AM on September 3, 2026: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

willcl-ark commented at 12:24 PM on September 3, 2026: member

Scanning the last 90 days (where logs still exist) in hebasto's repo I really only see this one where the vm failed to boot: https://github.com/hebasto/bitcoin-core-nightly/actions/runs/29139756629/job/86510586228#step:2:188

There are a handful of "random" functional test issues, which could be attributable to the vm-in-vm setup, but I'd recommend disabling those few tests. There are a number of legitimate functional test failures repeated over many days (e.g interface_gui.py) and few semi-legitimate-looking errors which could have been vm-flakiness.

Including the ~legit test timeout-type failures, but excluding real test failures by my count I see 18 "flaky failures" out of 812 jobs over the 90 days, which or 2.2%.

as long as the false positive rate is higher than the true positive rate, it doesn't seem worth it overall.

Our CI jobs have between a 7 and 2% fail rate (assuming all current failures are true-positives), per our CI job statistics.

So for references 2.2% as a potential false-positive rate is ~ in the same ballpark as the true-failures we see on jobs today (assuming all are true-failures). I agree a flakiness ~= to the bug-catching ability of non-flaky tests isn't great.

If we are more happy to backport fixes from broken nightly/master builds than run CI jobs here as we are worried they'll be too flaky (which is more like a regular CI/CD process to be honest) then that works for me too.

I'll leave this in draft for a few days in case anyone else wants to weigh in, before closing.

hebasto commented at 12:41 PM on September 3, 2026: member

Concept ACK on having at least one CI job that runs tests on a *BSD platform.