Significantly improves loadtxoutset performance via a number of changes:
Also adds a subcommand to bitcoin-util for getting a snapshot's hash_serialized_3, muhash, coin count and total amount values without importing to leveldb first.
On my systems, performance ends up bottlenecked by leveldb CPU usage.
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--006a51241073e994b41acfe9ec718e94-->
<!--021abf342d371248e50ceaed478a90ca-->
See the guideline and AI policy for information on the review process. A summary of reviews will appear here.
<!--174a7506f384e20aa4161008e828411d-->
Reviewers, this pull request conflicts with the following ones:
-Wunused by fanquake)CAmount a class by hodlinator)If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
<!--5faf32d7da4f0f540f40219e4f7537a3-->
Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):
src/test/validation_chainstatemanager_tests.cpp: cache.AddCoin(item.outpoint, Coin{item.coin}, false)<sup>2026-10-28 02:47:34</sup>
The performance change I see on a Ryzen 9 3900X system with RAID HDD storage is ~11minutes with to ~1m35s for a load, consisting of ~1m15s of CPU time followed by ~20s of disk flushing; with SSD and a slightly faster CPU, total time is reduced from ~5m30s to ~50s (no significant flush delay after the CPU work is completed here). These numbers are with the block 965000 utxo set.
<!--85328a0da195eb286784d51f73fa0af9-->
🚧 At least one of the CI tasks failed.
<sub>Task tidy: https://github.com/bitcoin/bitcoin/actions/runs/39762952091/job/119771350806</sub>
<sub>LLM reason (✨ experimental): CI failed because clang-tidy flagged a mismatched argument comment (moveout instead of moveto) in test/coinsviewoverlay_tests.cpp; warnings are treated as errors.</sub>
<details><summary>Hints</summary>
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
</details>
Can you please give us more context? Why do you consider an 11-minute load too slow? Is there a reason to think this is what's preventing people from using it - enough to justify yet more complexity in the most "load-bearing" parts of the code? Or is this just a demo to explore the possibilities?
Over the past years we've discussed how relying on someone else's UTXO set makes us uncomfortable (see #32377 and #36598). This change adds even more AssumeUTXO-specific complexity, which is exactly why we're considering removing it completely. I wouldn't object to adding it back later through IPC and the Kernel if there's still demand and doesn't add complexity - but there's no indication of that, especially since #36476 (possibly combined with SwiftSync) would already speed up actual UTXO set construction by an order of magnitude on resource-constrained nodes. Quickly loading someone else's UTXO set could still be a useful academic pursuit, but many of us are not comfortable polluting the code with multiple chainstates, which make consensus-critical paths harder to reason about.
Can you please give us more context?
In bips#2137 Sjors suggested splitting up the snapshot by coin to improve performance, which didn't seem sensible to me.
Why do you consider an 11-minute load too slow?
I don't consider it too slow; though it's possible it could be on slower systems than the ones I test on. But the reason it is that slow is because the database gets compacted multiple times, and it also seems to use excessive amounts of memory while loading, and there isn't really a good reason to do that. The difference between checking whether better performance is possible and actually making it available to users isn't that large.
This change adds even more AssumeUTXO-specific complexity, which is exactly why we're considering removing it completely.
As I've said on the list, I think the assumeutxo approach is a much better approach to resolving the "fast startup of a new node" challenge than trying to get a few extra percent out of IBD performance (which introduces its own large sets of complexities). If it's been decided that that's not one of this project's goals, that's fine I guess, but I haven't seen a project statement along those lines.
Making assumeutxo a reasonable default, rather than something for experts, does require significantly more complexity (ie, the various stuff for the BIP above, cf #35054). Likewise if we want to do FEC encoding so pruned nodes can help share utxo snapshots or support IBD.
<!--85328a0da195eb286784d51f73fa0af9-->
🚧 At least one of the CI tasks failed.
<sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/40163823354/job/121080686219</sub>
<sub>LLM reason (✨ experimental): CI failed because the IWYU check modified includes in src/node/utxo_snapshot.cpp, leaving a diff that caused the check to exit with code 1.</sub>
<details><summary>Hints</summary>
Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:
Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.
Leave a comment here, if you need help tracking down a confusing failure.
</details>
The snapshot file format is not node state: reading a snapshot needs no
chainstate, so move it out. This prepares the way for adding snapshot
parsing to bitcoin-util in a future commit.
Mostly move-only. The commit can be reviewed with the
--color-moved=dimmed-zebra option for an easier review.
SnapshotMetadata took the expected network in its constructor and compared
it while parsing, throwing on mismatch. Split that check out as a separate
`CheckNetwork` method, so that callers have the option to read snapshots
for arbitrary networks.
Also replace the per-object `const std::set` for checking supported
versions with an inline function.
Normally, we want updates to the database to be fsynced to disk so that
the database is in a consistent state even after a crash. This adds an
option to drop that guarantee, in favour of faster bulk writes, relying
on the caller to make other arrangements to ensure the resulting database
is flushed to disk.
This populates a chainstate leveldb (as a CDBWrapper) directly from a
batch (generated from Coin objects). This avoids the overhead of going
through a CoinsView map, and allows splitting the batch creation and
the batch write/flush across different threads.
Multi-producer, multi-consumer thread-safe queue with maximum capacity
for handing out jobs to threads.
Allows importing snapshots directly into leveldb (dbwrapper), without
passing through Chainstate or CoinsView first. Supports multithreaded
processing, with one thread reading, one thread hashing, one thread
serializing for the database, and one thread populating the database.
Adds an additional check that the coins in the snapshot are sorted in
txid order.
Note that because v2 dump follow the database key order (which encodes
output indices as VARINTs), but the UTXO commitment requires outputs
to be ordered numerically by output index, a sorting step is required
(eg VARINT(16512) sorts less than VARINT(16511). We bound the total
serialized transaction outputs by the consensus block weight limit to
bound the amount of date being sorted in case of corrupt snapshot files.
Nothing calls it yet, so behaviour is unchanged.
Co-Authored-By: Sjors Provoost <sjors@sprovoost.nl>
ActivateSnapshot and, therefore, the loadtxoutset RPC now use the new
utxo_snapshot.cpp pipeline.
The dump is no longer staged in a CCoinsViewCache and then hashed and
flushed out of it, so `-dbcache` worth of cached coins during a mainnet
load are gone along with the flush cycles. In addition, the hashing
of the dump takes place in parallel with the import, rather than after
it completes.
The old loader is left in place until the next commit removes it;
nothing calls it any more.
After the previous two commits, this code is no longer used, so remove it.
This also removes the circular dependency between node/utxo_snapshot
and validation.
Adds a subcommand for quickly generating the summary of a serialized
utxo set for verification purposes (including various hashes, coin count
and value).