node: add block template manager and track waitNext fee inflow #35581

ismaelsadeeq commented at 10:50 AM on June 22, 2026: member

Motivation

The mining interface and rpc helpers (WaitAndCreateNewBlock and friends) are free functions scattered across src/node/miner.cpp, and every caller has to pass ChainstateManager and KernelNotifications into them, which is verbose. SubmitBlock spins up a short-lived CValidationInterface subscriber on every call just to read back the block validation state. This PR adds a BlockTemplateManager that holds that state and exposes those helpers as methods, so callers don't have to thread chainman/notifications (or a throwaway subscriber) around.

It also removes a redundant pattern in waitNext. Currently, to decide whether mempool fee inflow rose enough to be worth a new template, waitNext locks cs_main and assembles a full block template every tick just to sum its fees and compare. After cluster mempool see https://delvingbitcoin.org/t/determining-blocktemplate-fee-increase-using-fee-rate-diagram/2052, we shouldn't have to build a whole block template to answer that question. This PR uses the #34803 mempool notifications to record the previous template state and track the inflow above the lowest included chunk, and only assembles and returns a new template once that inflow is above the threshold.

I added a fuzz harness and unit tests for the manager. The fuzz test already caught a bug in the miner where it skips legitimate transactions due to an incorrect block chunk size limit check #35580.

Changes

Populate the tx refs in the fee rate diagrams
Cache fee rate diagrams as chunks in ChangeSet and add SnapshotDiagrams
Add a MempoolUpdated notification that returns the before and after diagrams
Track the selected chunk information after each mempool update
Add fee-inflow staleness detection per template
Introduce a BlockTemplateManager wrapper around BlockAssembler that holds state and creates templates, and move the free functions (mining args, SubmitBlock, GetTip, WaitTipChanged, CooldownIfHeadersAhead, WaitAndCreateNewBlock) onto it as methods
Use the tracked inflow in waitNext instead of rebuilding a template each tick
Route in-process RPC and test template creation through BlockTemplateManager instead of the IPC Mining interface, which also drops the per-access block copies the IPC accessors (getBlock) forced
Add a fuzz test and unit tests

There are now two entry points for template creation. Mining clients go through the IPC Mining interface and get tracked templates, while in-process RPC and tests call BlockTemplateManager directly and get untracked ones. This is deliberate: one-shot RPC templates don't need staleness tracking, and it also avoids process-static RPC caches owning BlockTemplateImpl objects whose destructor reaches back into NodeContext at shutdown.

50% of the code addition here are tests ~1090 lines

DrahtBot commented at 10:50 AM on June 22, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35581.

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
Concept ACK	w0xlt, pablomartin4btc

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#35675 (mining: add block template manager by ismaelsadeeq)
#35671 (mining: add TxCollection to bandwidth-efficiently validate external block templates by Sjors)
#35646 (RFC: Separate out runtime errors from BlockValidationState using util::Expected by yuvicc)
#35580 (bugfix: compare non-adjusted chunk weight against block weight limit by ismaelsadeeq)
#35482 (fuzz: exercise the transaction-handling path in process_message(s) by HowHsu)
#35474 (node: move index ownership to NodeContext by w0xlt)
#34672 (mining: add reason/debug to submitSolution and unify with submitBlock by w0xlt)
#34565 (refactor: extract BlockDownloadManager from PeerManagerImpl by w0xlt)
#34075 (fees: Introduce Mempool Based Fee Estimation to reduce overestimation by ismaelsadeeq)
#33922 (mining: add getMemoryLoad() and track template non-mempool memory footprint by Sjors)
#32468 (rpc: generateblock to allow multiple outputs by polespinasa)
#31117 (miner: Reorg Testnet4 minimum difficulty blocks by fjahr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label CI failed on Jun 22, 2026

DrahtBot commented at 12:06 PM on June 22, 2026: contributor

🚧 At least one of the CI tasks failed. <sub>Task test ancestor commits: https://github.com/bitcoin/bitcoin/actions/runs/27947406418/job/82695559978</sub> <sub>LLM reason (✨ experimental): CI failed because the fuzz target (src/test/fuzz/blocktemplatemanager.cpp) would not compile: error: unknown type name 'SteadyClockContext'.</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

ismaelsadeeq force-pushed on Jun 22, 2026

w0xlt commented at 8:50 PM on June 22, 2026: contributor

Concept ACK

ismaelsadeeq force-pushed on Jun 23, 2026

DrahtBot removed the label CI failed on Jun 23, 2026

DrahtBot added the label Needs rebase on Jun 23, 2026

ismaelsadeeq force-pushed on Jun 24, 2026

DrahtBot removed the label Needs rebase on Jun 24, 2026

Sjors commented at 2:59 PM on June 25, 2026: member

It would be nice to have a PR focussed on the BlockTemplateManager:

08181220b6 node: introduce block template manager
8a4363ce6e test: add block template manager fuzz test
595acb3762 interfaces: create block template via block template manager
cbc15a02e2 node: move mining_args to block template manager
f7f496d8dc miner: move SubmitBlock into BlockTemplateManager
9099f6d875 node: move tip and wait helpers into BlockTemplateManager
dcbd4dbeb6 rpc, test: build block templates via BlockTemplateManager

I should be able to build #33922 / https://github.com/Sjors/bitcoin/pull/120 on top of that.

The waitNext() cluster mempool enhancement would be a natural followup.

in src/node/block_template_manager.cpp:234 in 8c73017dab

 229 | +                                   CAmount fee_threshold) const
 230 | +{
 231 | +    LOCK(m_mutex);
 232 | +    const auto snapshot = m_template_snapshots.find(template_id);
 233 | +    if (snapshot == m_template_snapshots.end()) return false;
 234 | +    if (snapshot->second.options != FlattenMiningOptions(options)) return false;

pablomartin4btc commented at 12:52 PM on July 1, 2026:

IsStale is a public method but has an implicit precondition: options must match the ones used when template_id was created, otherwise it silently returns false. Could a comment document this (if appropriate)?

in src/node/block_template_manager.cpp:341 in 8c73017dab outdated

 336 | +    bool& interrupt_wait,
 337 | +    uint64_t& new_template_id)
 338 | +{
 339 | +    auto now{NodeClock::now()};
 340 | +    const auto deadline = now + options.timeout;
 341 | +    const MillisecondsDouble tick{1000};

pablomartin4btc commented at 1:11 PM on July 1, 2026:

There was a comment there in the old code that was removed as I think part of it was no longer accurate. Perhaps it can be add:

// Alternate waiting for a new tip and checking if fees have risen.
// The staleness check is cheap but template creation is not, so we only rebuild once per second.

ismaelsadeeq commented at 11:07 AM on July 7, 2026:

Fixed in #35675

pablomartin4btc commented at 1:37 PM on July 1, 2026: member

Concept ACK - first time reviewing this area.

I can see this PR is based on top of #34803 (first 7 commits). The approach of moving free functions onto a class that holds state plus incremental snapshot tracking to avoid full template rebuilds seems clean. The fuzz test catching an actual bug (#35580) is a strong signal.

Left a few comments.

ismaelsadeeq commented at 3:46 PM on July 1, 2026: member

Thanks for taking a look at this @pablomartin4btc, I will address these comments when I open up a PR that adds the block template manager alone, as @Sjors suggested #35581 (comment).

ismaelsadeeq marked this as a draft on Jul 7, 2026

ismaelsadeeq force-pushed on Jul 7, 2026

DrahtBot added the label CI failed on Jul 7, 2026

DrahtBot added the label Needs rebase on Jul 7, 2026

node: introduce block template manager

Add BlockTemplateManager as an encapsulation
that wraps BlockAssembler::CreateNewBlock().

Add a simple unit test that verifies that block
template can be created via the block template manager.

97938177cd

node: move mining_args to block template manager

Pass the parsed mining args to BlockTemplateManager at construction and
expose them via GetInitBlockCreateOptions(), so the manager owns the
init-time block create options instead of NodeContext.

463901142b

miner: move SubmitBlock into BlockTemplateManager

Move SubmitBlockStateCatcher and SubmitBlock from miner.cpp into
BlockTemplateManager as a member function. This groups block submission
with block creation in the same class. The function uses m_chainman
directly instead of taking it as a parameter.

6db9b4766a

node: move tip and wait helpers into BlockTemplateManager

Move mining tip lookup and block-template waiting helpers into
BlockTemplateManager so the manager owns the template waiting flow,
including the notification wait helpers used by waitNext.

d4d97459c9

interfaces: create block template via block template manager

Route the Mining interface's createNewBlock() through
BlockTemplateManager::CreateNewTemplate() instead of constructing a
BlockAssembler directly.

425f2e82fc

rpc, test: build block templates via BlockTemplateManager

Use BlockTemplateManager directly for in-process RPC and test block
template creation instead of going through the IPC Mining interface.

This keeps RPC-created templates untracked and prevents process-static RPC
caches from owning BlockTemplateImpl objects whose destructor reaches back
into NodeContext during shutdown.

7004a1ac17

test: add block template manager fuzz test

Add a fuzz target that creates block templates through
BlockTemplateManager under fuzzed mining options and asserts a template
is always produced.

7d874095aa

txgraph: populate chunk refs in `GetMainStagingDiagrams` result

GetMainStagingDiagrams currently returns a pair of vector<FeeFrac>
containing only chunk feerates. Some callers need to know which
transactions belong to each chunk in order to compute a chunk hash.

Introduce TxGraph::Chunk, which pairs a chunk feerate with the refs of
all transactions in that chunk. Update GetMainStagingDiagrams to return a
pair of vector<TxGraph::Chunk>, and rename AppendChunkFeerates to
AppendChunks to reflect the richer return type.

Add SanityCheck coverage for the new behaviour.

31f65ac0e8

refactor: move-only: split GetPackageHash dbdff3adcc

mempool: cache fee rate diagrams as chunks in ChangeSet

CalculateChunksForRBF extracts only the FeeFrac from each TxGraph::Chunk
and discards the rest. Cache the fee rate diagram chunks in ChangeSet for
use in subsequent commits.

26001a065a

mempool: add SnapshotDiagrams to ChangeSet

Extract SnapshotDiagrams from CalculateChunksForRBF so the diagram
snapshot can be taken on removal paths that do not go through the RBF
codepath.

The const fix to RemoveStaged is required because GetRemovals()
returns a const reference.

35b5ebe6e9

refactor: move-only: extract dependency addition into a separate method 5673157131

mempool: add and fire MempoolUpdated signal on each mempool update path

Add MemPoolChunk and MemPoolChunksUpdate structs to
kernel/mempool_entry.h and MakeMemPoolChunks helper to txmempool.cpp
for converting TxGraph::Chunk vectors into MemPoolChunk vectors.

Each removal path now captures the before/after fee rate diagram chunks
and fires MempoolUpdated so subscribers can observe every mempool change.

This signal allows subscribers to observe mempool fee rate diagram
changes in an asynchronous way without holding the mempool lock.

This changes the order of operations in removeForBlock: all block
transactions in the mempool are now removed before conflict
removals.

In blockencodings_tests, SyncWithValidationInterfaceQueue() drains the
async MempoolUpdated signal queue before LOCK2(cs_main, pool.cs).
The LOCK2 moves after the sync to avoid deadlock.

667a35361e

test: add unit test for mempool update validation events 9ee0bd722a

miner: track selected chunks in block templates

Replace `m_package_feerates` with `m_template_chunks`, which records
each selected chunk's `FeePerWeight` feerate, unadjusted weight, sigops
cost, and the witness IDs of the transactions selected in that chunk.

This gives block templates enough information to describe both the
selection order and the transaction group selected at each step.

f22cc6a674

node: add block template snapshot tracking

Index template chunks by hash and feerate to update
snapshots without sorting them after each mempool addition.

4c6af5e055

node: update template snapshots from mempool callbacks

Register BlockTemplateManager for validation callbacks
and keep tracked snapshots synchronized with mempool and
chain-tip changes.

f2acefe725

node: add block template staleness detection

Track template fee inflow and report staleness once the fee improvement
reaches the requested threshold. The fee improvement is the lower-bound
difference between the current tracked chunk fees and the original
template fees.

a1c67245db

miner: use tracked fee inflow for waitNext

Use BlockTemplateManager snapshot tracking to
decide when waitNext should return a replacement
template due to fee inflow.

This avoids rebuilding full templates on every wait
tick just to compare fees. Tracked snapshots are cleaned
up by BlockTemplateImpl destruction, so an original template
can remain alive and call waitNext again after a replacement is returned.

828a7e3528

ismaelsadeeq force-pushed on Jul 8, 2026

DrahtBot removed the label Needs rebase on Jul 8, 2026

DrahtBot removed the label CI failed on Jul 8, 2026