Add output script index and corresponding script history RPC #36629

andrewtoth commented at 5:19 PM on October 23, 2026: contributor

Add a new index txoscriptindex which indexes all output scripts. Introduce a new getscripthistory RPC that returns all transactions that create outputs corresponding to the provided scripts, and optionally the transactions that spend those outputs if txospenderindex is also configured.

This has been a long-requested feature, attempted in previous PRs (#2802, #3652, #6835, #10370, #14053). Some concerns raised were about the ability for an index like this to scale. I think the new techniques being used to build the indexes (https://github.com/bitcoin/bitcoin/pull/35531, #36002) alleviate this concern. This index can be synced in a few hours and the current disk usage is ~52 GB. This is less than the ~66 GB txindex disk usage in v31 and earlier. This index can also be used with a pruned node, so a full node can be supported in < 100 GB of disk space. Missing blocks are returned by hash and can be requested from peers using getblockfrompeer.

Some other previous concerns were more conceptual, that this type of index could encourage address reuse and chain analysis. IMO this ship has sailed, and the current status quo of not shipping an address index is turning off users who expect this functionality. Workarounds involve installing, configuring, and syncing other third-party software, making the UX of using Bitcoin Core in a self-sovereign way unnecessarily difficult.

The index hashes each output script with SHA256, then applies salted SipHash-1-3-UJ and stores a five-byte prefix. This is to support the popular Electrum protocol, which supplies only the script hash. The RPC can also look up by address or descriptors. Ranged descriptors are also supported, so a single RPC call can return the entire history of a receive/change xpub pair.

Because there are some scripts with millions of outputs, we introduce a new OrderedVarInt which serializes the block height after the script hash prefix. This lets LevelDB seek directly to the prefix + block height, so we can page by block height range.

To prevent DoS concerns by supplying an address with millions of outputs, the RPC tracks a byte budget of reads and throws immediately if too many reads are attempted. The error message instructs the user to narrow the block range.

Improvements to txospenderindex (https://github.com/bitcoin/bitcoin/pull/36480) will also allow us to use that pruned. The end result would allow users to run with all optional indexes (including this one) in < 200 GB of disk space. Of course a larger pruned setting would be desired if many historical block lookups were being requested.

DrahtBot commented at 5:19 PM on October 23, 2026: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

External sites

Code coverage & benchmarks: https://corecheck.dev/bitcoin/bitcoin/pulls/36629
Ralph review <img src=https://review.fish.foo/assets/bitcoin/bitcoin/36629.png alt="Ralph review status"> https://git.fish.foo/bitcoin/bitcoin/pulls/36629#author-ralph

Reviews

See the guideline and AI policy for information on the review process.

Type	Reviewers
ACK	kravens
Concept ACK	sedited, l0rinc, w0xlt, romanz, ViniciusCestarii, xyzconstant

If your review is incorrectly listed, please copy-paste <code></code> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#36618 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36618.svg"> (test: Wait for index sync in wallet_fast_rescan.py by maflcko)
#36603 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36603.svg"> (Improve loadtxoutset performance by ajtowns)
#36598 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36598.svg"> ([Demo] validation: remove AssumeUTXO by l0rinc)
#33117 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/33117.svg"> (Interfaces: Expose UTXO Snapshot Loading and Add Progress Notifications by D33r-Gee)
#31252 <img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/31252.svg"> (rpc: print P2WSH and P2SH redem Script in getrawtransaction and getblock by polespinasa)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

src/init.cpp: std::make_unique<TxoScriptIndex>(interfaces::MakeChain(node), index_cache_sizes.txoscript_index, false, do_reindex)
src/test/txoscriptindex_tests.cpp (txoscriptindex_initial_sync): index.Find(script_hash, -1, 95, remaining_bytes)
src/test/txoscriptindex_tests.cpp (txoscriptindex_initial_sync): index.Find(script_hash, 95, 90, remaining_bytes)
src/test/txoscriptindex_tests.cpp (txoscriptindex_collisions_and_pruned_blocks): index.Find(script_hash, 0, 102, remaining_bytes)
src/test/txoscriptindex_tests.cpp (txoscriptindex_collisions_and_pruned_blocks): index.Find(script_hash, 0, 100, remaining_bytes)
src/test/txoscriptindex_tests.cpp (txoscriptindex_read_failures): index.Find(script_hash, 1, 2, remaining_bytes) (appears twice)
src/test/txoscriptindex_tests.cpp (txoscriptindex_false_positive_budget): index.Find(script_hash, 1, 2, remaining_bytes) (appears three times)
src/test/txoscriptindex_tests.cpp (txoscriptindex_false_positive_budget): index.Find(script_hash, 1, 1, remaining_bytes) (appears twice)
src/test/txoscriptindex_tests.cpp (txoscriptindex_large_transaction_count): index.Find(txoscriptindex::Sha256ScriptPubKey(script), 101, 101, remaining_bytes)
src/test/txoscriptindex_tests.cpp (txoscriptindex_reorg_erases_rows): index.Find(txoscriptindex::Sha256ScriptPubKey(script), 101, 101, remaining_bytes)

Possible places where comparison-specific test macros should replace generic comparisons:

src/test/ordered_varint_tests.cpp: The truncation-loop BOOST_CHECK_THROW(DecodeOrderedVarInt(...), std::ios_base::failure) → Prefer checking the expected failure reason with BOOST_CHECK_EXCEPTION.
src/test/ordered_varint_tests.cpp: The invalid-prefix BOOST_CHECK_THROW(DecodeOrderedVarInt(...), std::ios_base::failure) → Prefer BOOST_CHECK_EXCEPTION with the expected reason, "Invalid OrderedVarInt encoding".
src/test/ordered_varint_tests.cpp: The two BOOST_CHECK_THROW(EncodeOrderedVarInt(...), std::ios_base::failure) checks → Prefer BOOST_CHECK_EXCEPTION with the expected reason, "OrderedVarInt value too large".
src/test/ordered_varint_tests.cpp: The overflow BOOST_CHECK_THROW(overflow >> Using<OrderedVarIntFormatter>(decoded), std::ios_base::failure) → Prefer BOOST_CHECK_EXCEPTION with the expected reason, "OrderedVarInt exceeds limit of type".
src/test/txoscriptindex_tests.cpp: The invalid-encoding-loop BOOST_CHECK_THROW(invalid >> decoded, std::ios_base::failure) → Prefer checking the expected failure reason with BOOST_CHECK_EXCEPTION.
src/test/txoscriptindex_tests.cpp: BOOST_CHECK_THROW(DataStream{} << txoscriptindex::DBValue{}, std::ios_base::failure) → Prefer BOOST_CHECK_EXCEPTION with the expected reason, "Empty txoscriptindex value".
src/test/txoscriptindex_tests.cpp: BOOST_CHECK_THROW(oversized >> decoded, std::ios_base::failure) → Prefer BOOST_CHECK_EXCEPTION with the expected reason, "Invalid txoscriptindex transaction offset".
test/functional/rpc_getscripthistory.py: assert output["spent"] is spent → Prefer assert_equal(output["spent"], spent).

2026-10-29 14:31:32

andrewtoth force-pushed on Oct 23, 2026

DrahtBot added the label CI failed on Oct 23, 2026

DrahtBot commented at 5:41 PM on October 23, 2026: contributor

🚧 At least one of the CI tasks failed. Task ASan + LSan + UBSan + integer: https://github.com/bitcoin/bitcoin/actions/runs/39994088566/job/120552982572 LLM reason (✨ experimental): CI failed because ordered_varint_tests triggered UBSan for an implicit uint64_t-to-uint8_t truncation in index/ordered_varint.h:55.

<details><summary>Hints</summary>

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

</details>

DrahtBot removed the label CI failed on Oct 23, 2026

sedited commented at 6:54 PM on October 23, 2026: contributor

Concept ACK

l0rinc commented at 7:05 PM on October 23, 2026: contributor

Concept ACK

w0xlt commented at 8:00 PM on October 23, 2026: contributor

Strong Concept ACK

kravens commented at 10:56 PM on October 23, 2026: contributor

Concept ACK.

Reviewed 614302e7d14a16fe28a2e8945fa99165693fcf94. I built it and ran txoscriptindex_tests, baseindex_tests, rpc_getscripthistory.py, rpc_misc.py, feature_init.py and feature_assumeutxo.py, all passing. I also did a manual regtest run: address, Electrum scripthash, ranged wpkh() descriptor, verbosity 0/1/2, and spender linking. Results matched expectations. My comments are mostly about structure:

1. The lookup budget does not count block and undo reads. At verbosity 2, tx_to_json reads the full block and its undo data for every block that holds a non-coinbase match (rpc/blockchain.cpp:4116). Step 10 also reads a full block whenever two or more extra spenders share one (rpc/blockchain.cpp:4064). Neither read is charged. I ran this on regtest with 150 matches, each in its own block padded to ~60 kB. Bytes read by bitcoind during one call (rchar):

verbosity	matching tx bytes	bytes read by node
0	52,039	0.94 MB
2	52,039	21.2 MB

On mainnet that means one full block plus its undo data per matching block. A query that fits in the 50 MiB budget can therefore read far more than 50 MiB. Either charge these reads, or say in the help text that the budget does not cover them.

2. OrderedVarInt could be BigEndianFormatter<3>. Big-endian already sorts by height, and the same key already uses it for the tx offset. Both encodings are 3 bytes for heights 16,512 through 2,113,663 (roughly until 2048). After that, BE3 stays at 3 bytes while OrderedVarInt grows to 4, and BE3 goes up to height 16,777,215. I tried it locally. It removes ordered_varint.h and its tests (+4/−210), and the PR's unit and functional tests still pass:

+    static constexpr uint32_t HEIGHT_SIZE{3};
 ...
-                  Using<OrderedVarIntFormatter>(obj.height),
+                  Using<BigEndianFormatter<HEIGHT_SIZE>>(obj.height),

3. Ordering spenders doesn't need a block read. txospenderindex keys already store a CDiskTxPos, and its nTxOffset grows with the transaction's position in the block. I added it to TxoSpender and sorted extra spenders by (height, tx_offset). That removes the grouping loop and the GetBlockChecked call (+6/−20). test_confirmed_transaction_order still passes. It fails if the offset is dropped from the comparison, so the test does cover this.

4. Find could take the whole set of script hashes. Today the RPC calls Find once per hash and merges results by (height, tx_offset). A transaction paying two scripts of one descriptor is read and charged twice; rpc_getscripthistory.py relies on this ("repeated reads of overlapping transactions exceed the work budget"). A multi-hash Find would read each candidate once and drop both ScriptHistoryMatch::tx_offset and the by_position merge. The budget would then trip later than it does now, never earlier, so that test case would need updating. Returning pruned blocks as their own set instead of null-tx placeholders would also remove the duplicated branch at index/txoscriptindex.cpp:163 and :174, and the matching filter in the RPC.

5. One budget, one error. Index lookups charge a uint64_t& and fail with "Script history index lookup limit exceeded…". Extra transaction bodies throw a std::runtime_error with "Script history exceeds the … byte budget" (rpc/blockchain.cpp:3933). Both enforce the same limit through two code paths with two messages. The index-layer message also names RPC options (index/txoscriptindex.cpp:138). A small budget type that both layers charge, with the user-facing wording kept in the RPC, would be simpler.

6. Could the RPC go in its own file? blockchain.cpp grows by ~530 lines to 4,238. The handler is one ~400-line lambda that opens with a 12-step index comment. src/rpc/scripthistory.cpp with one function per step would be easier to read and review.

7. TxToJSON is now declared in rawtransaction_util.h. That header belongs to bitcoin_common, which the wallet also links. TxToJSON takes a Chainstate and is still defined in rawtransaction.cpp in bitcoin_node. Declaring it in a node-side header would keep that boundary clean.

Nits:

The mempool scan matches against scripts or script_hashes depending on how the query was given (rpc/blockchain.cpp:3986). Hashing everything up front would leave one set and remove the branch.
When range is given, the descriptor is parsed twice: once at rpc/blockchain.cpp:3896 and again inside EvalDescriptorStringOrObject.
RawTransactionDoc changes the getrawtransaction help text:
- blockhash moves below vsize_adjusted and gains "(omitted for mempool transactions)".
- The hex description changes.
- At verbosity 2, fee moves to the end.
Probably fine, but worth a line in the commit message.

Disclosure: AI (Claude Opus 5.5) was used to find these issues and to draft this review.

serialize: add OrderedVarIntFormatter f8a98778bb

andrewtoth force-pushed on Oct 25, 2026

in src/index/txoscriptindex_key.h:28 in a221edb4cb

  23 | + *
  24 | + *   ['s', hash prefix, height] -> transaction offsets
  25 | + *   ["spk_hash_salt"]          -> hasher salt
  26 | + *   ['B']                     -> sync locator
  27 | + *
  28 | + * hash prefix is SipHash-1-3-UJ of SHA256(scriptPubKey), truncated to 5 bytes.

romanz commented at 8:24 AM on October 26, 2026:

SHA256(scriptPubKey)

Electrum 1.7.x protocol is now using scriptPubKey-based subscription and queries, so maybe it would simpler to avoid SHA256 here?

andrewtoth commented at 2:32 PM on October 29, 2026:

I'm not sure it will make it much simpler. I think it's worth it to keep this for backwards compatibility.

romanz commented at 6:42 PM on October 26, 2026: contributor

Concept ACK, thanks!

in src/rpc/blockchain.cpp:3741 in cf9b981fcc

3736 | +    }};
3737 | +
3738 | +    return RPCMethod{
3739 | +        "getscripthistory",
3740 | +        "Returns transactions that create outputs matching a scriptPubKey, plus any transactions that spend those outputs.\n"
3741 | +        "The script may be specified as an address, an output descriptor, or SHA256 of the scriptPubKey (hex-encoded in Bitcoin byte order, same as Electrum scripthash).\n"

ViniciusCestarii commented at 7:23 PM on October 26, 2026:

In "rpc: add getscripthistory" cf9b981fcc7ae52c7864beae35df5d60b5c322b4

nit: getscripthistory help writes "Bitcoin byte order" which doesn't seem to be used elsewhere in the repo. I believe "reversed byte order" is a better fit

in src/rpc/blockchain.cpp:3752 in cf9b981fcc outdated

3747 | +        "A block height reported in a lookup-limit error is a retry hint, not a continuation point; no partial history is returned.\n"
3748 | +        "Spentness is taken from the UTXO set and optionally the mempool when include_mempool=true. Confirmed spending transactions that do not themselves contain scriptPubKey matches are only returned when -txospenderindex is synced and include_spending_transactions=true.\n\n"
3749 | +        "If verbosity is 0 or omitted, each transaction contains the serialized transaction as hex, plus spentness for matching outputs.\n"
3750 | +        "If verbosity is 1, each transaction is a JSON Object with the same fields as getrawtransaction verbosity 1, plus spentness for matching outputs.\n"
3751 | +        "If verbosity is 2, each transaction is a JSON Object with the same fields as getrawtransaction verbosity 2, plus spentness for matching outputs.\n"
3752 | +        "Mempool transactions omit blockhash, matching getrawtransaction.",

ViniciusCestarii commented at 7:27 PM on October 26, 2026:

In "rpc: add getscripthistory" cf9b981fcc7ae52c7864beae35df5d60b5c322b4

nit: getscripthistory help could say pruned_blocks blocks can be fetched with getblockfrompeer and then retry. It also could also state that pruned_blocks can contain false positives (blocks where no tx match the output script).

ViniciusCestarii commented at 10:06 PM on October 26, 2026: contributor

Concept ACK. I skimmed the code but plan to take a deeper look. Commented 2 nits.

index: add txoscriptindex f9d4d0af81

index: expose spender transaction offsets bf0377cc82

rpc: share transaction JSON helpers 0843f7b5c9

andrewtoth force-pushed on Oct 29, 2026

rpc: add getscripthistory 1fc5b618e3

andrewtoth force-pushed on Oct 29, 2026

andrewtoth commented at 2:31 PM on October 29, 2026: contributor

Thanks @kravens, @romanz, and @ViniciusCestarii for the reviews, and @sedited, @l0rinc, and @w0xlt for the concept ACKs!

I took some of the suggestions, and made some additional updates. This changes the index format so anyone who was testing with the previous version needs to rebuild it.

More compact index layout for duplicate scripts in the same block. Instead of one row per output, we group all identical outputs in the same block and serialize the offsets between each of them as the value corresponding to a single key for that script and block height.
Increase the prefix size from 5 bytes to 6 bytes. While 5 bytes is sufficient for the txindex, a collision in this index has a much worse case. If a script with none or a few outputs being looked up collides with an output that has millions of outputs, it will require scanning through all the block ranges and reading all the false positive outputs.
Added an asof optional parameter to fix a blockheight when paging through requests. This let's a caller make repeated queries that set the spentness of outputs based on the asof block.
Bound the undo and block reads to the lookup byte budget and throw if exceeded.
Expose tx_offset from txospenderindex and sort spenders with it.
Moved TxToJSON to rpc/rawtransaction.h
Various documentation changes, including release notes.
Added more test coverage to kill some mutants.

DrahtBot added the label CI failed on Oct 29, 2026

xyzconstant commented at 4:53 PM on October 29, 2026: contributor

Concept ACK

kravens commented at 6:10 PM on October 29, 2026: contributor

tACK 1fc5b618e3693c824124dd4ee4c3c95e100e645d

Re-reviewed the changes since 614302e7 (same base, no rebase). Built it and ran txoscriptindex_tests, txospenderindex_tests, ordered_varint_tests, caches_tests, baseindex_tests, rpc_getscripthistory.py, rpc_misc.py, rpc_gettxspendingprevout.py, feature_init.py and feature_assumeutxo.py. All pass.

From my earlier comments, 1 (block and undo reads outside the budget), 3 (spender ordering without a block read) and 7 (TxToJSON header) are fixed: tx_to_json now charges the serialized undo and block (rpc/blockchain.cpp:4154, :4159), test_undo_size_limit covers the limit, and the step-10 block read is gone. 2, 4, 5 and 6 were suggestions; no objection to leaving them.

What I checked in the new layout:

One row per (prefix, height); the value is a run of VARINT offset deltas read until the stream is empty. Serialize rejects empty, non-increasing and out-of-block offsets, and Unserialize caps the count at MAX_BLOCK_WEIGHT / MIN_TRANSACTION_WEIGHT. Keys, values and transaction bodies are all charged to the budget.
Find keeps one block file open across rows and seeks with SEEK_SET, which keeps the AutoFile obfuscation position correct (streams.cpp:41). Matches still come back in (height, offset) order, so the by_position merge in the RPC holds.
CustomRemove erases the (prefix, height) key and a competing block at the same height overwrites it; txoscriptindex_reorg_erases_rows covers the rewind.
6-byte prefix: with roughly 2^31 distinct scripts on mainnet, the chance that a queried script shares its bucket with any other script drops from about 2^-9 to 2^-17. Agree with the change.

asof: I read it with the paging use case in mind and could not break it. Spentness at asof comes only from the matches and txospenderindex; spenders above asof are dropped by the ancestor check at rpc/blockchain.cpp:4097 and the output stays unspent. Reorgs above asof leave the answer unchanged, reorgs through asof fail the final check, and confirmations and times come from the asof block via TxToUniv, so a page stays identical while the tip moves. test_asof exercises all of this.

Observations, none blocking:

asof could be served during the initial index sync. BlockUntilSyncedToCurrentChain() returns false until the whole sync finishes (index/base.cpp:444), so a node still building either index answers nothing, even for an asof block both indexes passed long ago. The ancestor checks at rpc/blockchain.cpp:3954 and :3960 already guarantee that every row up to asof exists (the summary's best block lags the DB by up to 30 s during the initial sync and never leads it). Skipping the two BlockUntilSyncedToCurrentChain() calls when asof is given would let a server start paging old history while the index catches up. Fine as a follow-up.
Explicit include_mempool=true together with asof is silently ignored (rpc/blockchain.cpp:3890). The help text says so, but an RPC_INVALID_PARAMETER would make the conflict visible to a caller who set both on purpose.
Release notes. The summary mentions release notes, but the branch only touches doc/files.md. Did a doc/release-notes-36629.md miss the push?
rpc: share transaction JSON helpers still has an empty commit body. It changes the getrawtransaction help layout (blockhash moves below vsize_adjusted, fee moves last at verbosity 2). One line in the message would help whoever bisects a help-text diff later.
The "test ancestor commits" CI job died in p2p_segwit.py on an assertion that looks unrelated, before it reached the later commits, so the per-commit build check needs a re-run.

Disclosure: AI (Claude Fable 5.1) was used to find these issues and to draft this review.

DrahtBot requested review from romanz on Oct 29, 2026

DrahtBot requested review from ViniciusCestarii on Oct 29, 2026

DrahtBot requested review from sedited on Oct 29, 2026

DrahtBot requested review from l0rinc on Oct 29, 2026