mining: add TxCollection to bandwidth-efficiently validate external block templates #35671

Sjors commented at 7:36 PM on July 6, 2026: member

The Stratum v2 spec defines a Job Declarator Server (JDS), whose job it is to approve custom miner templates. It currently maintains its own mempool mirror, which it populates by repeatedly calling getBlock() and waitNext() and then requesting any missing transactions from the miner (since #34020 it can use getTransactionsByWitnessID() to first ask the node).

The new TxCollection interface introduced by this PR removes the need for such a mirror mempool. We build the block template ourselves now, based on the requested transaction list and our own mempool.

The following flow is demonstrated in https://github.com/stratum-mining/sv2-apps/pull/599:

Miner generates a template, e.g. using our createNewBlock() IPC method
They send the list of Wtxids to the pool's JDS
JDS calls TxCollection::collectTxs() (this PR) with that Wtxid list
We check the list against our mempool and hold a CTransactionRef for each match
The JDS queries unknownTxPos() to learn what's missing in our mempool
The JDS requests the fully serialized missing transactions from the miner (using native Stratum v2 messages)
JDS calls addMissingTxs() which completes the collection
JDS calls makeTemplate() which reconstructs the block and validates it. The JDS needs the check to succeed, may relay the BIP-22 failure reason to the miner.
It returns a BlockTemplate which works as if we generated the template ourselves, but with methods like waitNext() disabled. The JDS can call submitSolution() (the sv2 spec allows for redundant block reconstruction by both the miner node and the JDS).

If this sounds very similar to compact block relay: it is! In fact, it would be easy to expand the above methods to take transaction short ids instead of Wtxid. That may be useful for the p2pool revival project, which needs a way to validate and relay weak blocks as compact blocks. The p2pool client software would need to (trivially) verify the PoW, but could use the TxCollection interface to reconstruct and verify the block content. That said, the SRI JDS implementation is much further along and likely to be the first consumer of this.

Commits:

ipc: add TxCollection scaffold - step 3 and 4
mining: add TxCollection unknownTxPos - step 5
mining: add TxCollection addMissingTxs - step 7
mining: add coinbase transaction helper - we need a dummy coinbase to verify the block, so extract a helper from BlockAssembler::CreateNewBlock()
mining: make TxCollection create a BlockTemplate - step 8 and 9
mining: restrict externally generated templates - step 9

Potential followups:

have addMissingTxs() (optionally) insert favorable transactions into our mempool
add a data structure to store transactions that don't meet our mempool threshold (so TxCollection can grab them, instead of needing another round trip)

Conceptual review is welcome. I need to polish the commits a bit further, so it's better to hold off on that. I opened the PR now because the concept was brought up in #34020#pullrequestreview-4635352194.

Keeping this in draft until #34672 lands, to match its handling of the reason/debug fields.

Sjors commented at 7:37 PM on July 6, 2026: member

@plebhash it would be very useful to have a (very) rough JDS draft that uses this interface, to see if it actually gets rid of the mirror mempool.

DrahtBot commented at 7:37 PM on July 6, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35671.

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
Concept ACK	ismaelsadeeq

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#35675 (mining: add block template manager by ismaelsadeeq)
#35581 (node: add block template manager and track waitNext fee inflow by ismaelsadeeq)
#35569 (Encapsulation for CTransaction by purpleKarrot)
#33922 (mining: add getMemoryLoad() and track template non-mempool memory footprint by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Mining on Jul 6, 2026

Sjors force-pushed on Jul 6, 2026

DrahtBot added the label CI failed on Jul 6, 2026

DrahtBot commented at 7:51 PM on July 6, 2026: contributor

🚧 At least one of the CI tasks failed. <sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/28818181619/job/85462853775</sub> <sub>LLM reason (✨ experimental): CI failed because IWYU reported/required include fixes (modified files) and intentionally exited non-zero.</sub>

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

DrahtBot removed the label CI failed on Jul 6, 2026

ismaelsadeeq commented at 9:41 AM on July 7, 2026: member

Concept ACK

The JDS queries unknownTxPos() to learn what's missing in our mempool

This could be collapsed to the response "JDS calls TxCollection::collectTxs() (this PR) with that Wtxid list" to reduce the round trip?

Sjors force-pushed on Jul 7, 2026

Sjors commented at 10:49 AM on July 7, 2026: member

I added an optional coinbase argument to makeTemplate(). Based on my draft JDS implementation https://github.com/stratum-mining/sv2-apps/pull/599 this does appear useful, e.g. for checking the reward amount. We still generate a dummy if no coinbase transaction is provided. @ismaelsadeeq collectTxs() is the constructor so it needs to return an interface. I don't think it can return both the TxCollection interface and the list of missing transactions. But clients can chain requests (not sure if we've fully implemented the on the libmultiprocess side).

I can also imagine a (slight) benefit in clients being able to call unknownTxPos() multiple times, e.g. maybe in a future upgrade we could have the node request missing transactions from peers - not sure how useful that is though.

Sjors referenced this in commit 354f23abd0 on Jul 7, 2026

DrahtBot added the label Needs rebase on Jul 8, 2026

ipc: add TxCollection scaffold

Add the TxCollection interface and the CollectedTxs class that backs it.
The constructor looks up each requested wtxid in the mempool and keeps a
reference to any transaction that is already present, so later commits
can report which requested transactions are still missing and let the
client fill them in.

19bae00aa1

mining: add TxCollection unknownTxPos e7309ab7d2

mining: add TxCollection addMissingTxs

Now that the collection can be mutated after construction, guard the
collected transactions with a mutex: IPC clients may call TxCollection
methods concurrently from different threads.

ee1c43e3ce

mining: add coinbase transaction helper

Move the coinbase construction out of CreateNewBlock() into a reusable
BlockAssembler::CreateCoinbaseTx() helper, so a later commit can build a
coinbase when assembling a template from externally collected
transactions.

This does not change behavior. Review with --color-moved=dimmed-zebra.

1611aadbed

mining: make TxCollection create a BlockTemplate

Add TxCollection::makeTemplate(), which assembles the collected
transactions, in the requested order, into a block that builds on the
given prevhash, and validates it.

The block is validated by the same TestBlockValidity() call used by
checkBlock(), with the proof-of-work and merkle-root checks disabled.

A dummy coinbase is added only so the transactions can be validated as
part of a block. Its output pays no fees (fees=0), because the total fee
amount is not tracked here. Clients construct the real coinbase
themselves, which is why the next commit disables getCoinbaseTx() for
externally generated templates.

When the requested prevhash does not match the active tip, we mirror the
getblocktemplate proposal reasons: 'stale-prevblk' when prevhash is an
ancestor of the current tip, and 'inconclusive-not-best-prevblk' when it
is unknown or on a fork.

5fdd934905

mining: restrict externally generated templates

Disable methods that are not expected to be used in the externally
provided template use case.

getTxFees() and getTxSigops() could be implemented, but it's not
worth doing since the external software that constructed the
template is expected to know this.

getCoinbaseTx() is disabled because the dummy coinbase is only added to
allow validating the collected transactions as a block. It pays no fees
(fees=0) and should not be used by clients, which construct the real
coinbase themselves.

9bb89df02f

Sjors force-pushed on Jul 8, 2026

Sjors commented at 6:41 AM on July 8, 2026: member

Rebased after #34020:

first test commit no longer needed
bumped collectTxs() to @10

Sjors referenced this in commit 22bc165d93 on Jul 8, 2026

Sjors referenced this in commit 487f2e365b on Jul 8, 2026

Sjors referenced this in commit ddadd4bde7 on Jul 8, 2026

Sjors referenced this in commit f20d0ce98b on Jul 8, 2026

Sjors referenced this in commit 1d65d7bbf5 on Jul 8, 2026

Sjors referenced this in commit 31d3c510db on Jul 8, 2026

DrahtBot removed the label Needs rebase on Jul 8, 2026