net: serve `MSG_BLOCK` by stripping witnesses in place #36548

pull l0rinc wants to merge 8 commits into bitcoin:master from l0rinc:l0rinc/msg-block-serving changing 8 files +482 −48
  1. l0rinc commented at 1:08 AM on October 16, 2026: contributor

    Problem: Serving blocks to legacy nodes currently decodes and reserializes them just to remove witnesses. MSG_WITNESS_BLOCK already avoids this work by serving the stored bytes directly. Faster witnessless IBD, as proposed in #36476, also needs peers to serve these blocks efficiently.

    Fix: Read disk-backed, non-Signet MSG_BLOCK replies into an owned byte buffer, check the header's proof of work and expected hash, and strip witness ranges from that buffer in place.

    Extract a shared helper so decoding transactions and skipping their bytes follow the same encoding rules. Signet keeps ReadBlock for its solution check. Header checks now precede transaction parsing, so records corrupted in both places can log a different error.

    Performance:

    Time to read a block from disk and build its non-witness message:

    Block Before After Speedup
    block413567 1.31 ms 0.147 ms 8.9x
    Same block with synthetic witnesses 3.47 ms 0.374 ms 9.3x

    The synthetic fixture adds a 512-byte witness item and an empty item to every input.

    Related work: This extends the raw witness-block serving added in #13151. The #13011 discussion and the closed #13098 identify unnecessary transaction hashing when serving historical blocks. The implementation also builds on the bulk block reads introduced in #31551.

  2. test: cover block serving serialization
    Check exact `MSG_BLOCK` and `MSG_WITNESS_BLOCK` payloads before and after recent-block cache eviction.
    Cover successful and failing `ReadBlock` cases before introducing the serialized reader.
    98ce82547b
  3. refactor: share block read hash checks
    Extract the expected-hash check for reuse by the serialized reader.
    Keep Signet solution validation before the shared check.
    7c6e5549f5
  4. bench: measure non-witness block serving
    Measure disk reads followed by `TX_NO_WITNESS` message serialization before replacing the full block reader.
    
    Witness-free: 1.31 ms/block
    Witness-bearing: 3.47 ms/block
    e2b4f330cb
  5. node: read non-witness block payloads
    Add `ReadBlockWithoutWitness` as a decode-and-reserialize reference implementation.
    It checks proof of work and the expected header hash before transaction decoding, drops trailing bytes, and requires non-Signet consensus because it omits the Signet solution check.
    
    The benchmark takes 1.42 ms/block for witness-free data and 3.65 ms/block for witness-bearing data.
    9f15a0da42
  6. net: serve `MSG_BLOCK` from serialized data
    Use `ReadBlockWithoutWitness` for disk-backed `MSG_BLOCK` requests.
    Keep Signet and cached, witness, compact, and filtered block requests on their existing paths.
    f0d2fb57be
  7. refactor: extract transaction decoding rules
    Extract marker, optional-data, input, output, and witness handling into `UnserializeTransactionBody` so block witness stripping can reuse the decoder's validation.
    Keep the ordinary decoder's reads, clears, and error order unchanged.
    8e2e518d2a
  8. node: strip witnesses without decoding
    Walk serialized transactions with `UnserializeTransactionBody` and copy only retained ranges.
    This shares the ordinary decoder's marker, optional-data, CompactSize, and witness checks without allocating transactions or computing their cached hashes.
    Witness-free encodings are still walked completely, and trailing bytes remain excluded.
    
    The benchmark falls from 1.41 to 0.161 ms/block for witness-free data and from 3.65 to 0.384 ms/block for witness-bearing data.
    3557ab74e6
  9. DrahtBot added the label P2P on Oct 16, 2026
  10. DrahtBot commented at 1:08 AM on October 16, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    ACK fametrano

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36521 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36521.svg"></sub> (Keep witness stacks in serialized form in CScriptWitness by sipa)
    • #36074 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36074.svg"></sub> (scripted-diff: [test] Add util/check.h includes for assertions by maflcko)
    • #36000 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36000.svg"></sub> (validation: prefetch blocks while connecting by l0rinc)
    • #35731 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35731.svg"></sub> (Indexes: Harden the flush-error notification invariant by arejula27)
    • #32554 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/32554.svg"></sub> (bench: replace embedded raw block with configurable block generator by l0rinc)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

    LLM Linter (✨ experimental)

    Possible places where comparison-specific test macros should replace generic comparisons:

    • test/functional/p2p_getdata.py: assert len(from_binary(CBlock, full).serialize(with_witness=False)) < len(full) → Prefer assert_greater_than(len(full), len(from_binary(CBlock, full).serialize(with_witness=False))).

    <sup>2026-10-16 13:37:48</sup>

  11. DrahtBot added the label CI failed on Oct 16, 2026
  12. DrahtBot commented at 2:21 AM on October 16, 2026: contributor

    <!--85328a0da195eb286784d51f73fa0af9-->

    🚧 At least one of the CI tasks failed. <sub>Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/38833479928/job/116757258223</sub> <sub>LLM reason (✨ experimental): CI failed because IWYU modified includes in src/bench/readwriteblock.cpp, and the formatting check detected uncommitted changes.</sub>

    <details><summary>Hints</summary>

    Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

    • Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    • A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.

    • An intermittent issue.

    Leave a comment here, if you need help tracking down a confusing failure.

    </details>

  13. node: strip block witnesses in place
    Reuse the owned input buffer and move retained ranges only after a removed witness gap.
    Each destination is at or before its source, so `memmove` leaves unread bytes intact.
    Erase the unused tail after the walk.
    
    The benchmark falls from 0.161 to 0.147 ms/block for witness-free data and from 0.384 to 0.374 ms/block for witness-bearing data.
    0b01ce05e1
  14. l0rinc force-pushed on Oct 16, 2026
  15. l0rinc closed this on Oct 18, 2026

  16. l0rinc reopened this on Oct 18, 2026

  17. DrahtBot removed the label CI failed on Oct 18, 2026
  18. pmazzocchi referenced this in commit ca94fe44d6 on Oct 18, 2026
  19. fametrano commented at 11:41 AM on October 25, 2026: none

    tACK 0b01ce05e162ee61fb0e3c0d8413e0e1af4c1976

    I checked that the new MSG_BLOCK reply is byte for byte the old one. On 214 blocks, the head's ReadBlockWithoutWitness matches its ReadBlock plus TX_NO_WITNESS, that old path matches the base's, and btclib, a library I maintain, agrees. The blocks are mainnet 481824, 709632, 774628, 920000 and the bench's 413567, plus 209 regtest blocks with segwit v0 and v1 spends, 1,000 witness items, empty stacks and items, a 3,990,000-byte annex, 300 inputs, 316 outputs, 260 transactions, or no witness. Over P2P, a head node and a base node give the same MSG_BLOCK and MSG_WITNESS_BLOCK payloads for all 209: 208 from disk, and the two successive tips from the recent-block cache.

    Each of seven mutants of StripBlockWitness and ReadBlockWithoutWitness fails blockmanager_tests. They range from keeping the marker or the witnesses to skipping the proof-of-work or hash check. The new p2p_getdata case also passes against a base node, so it guards the payload format rather than the new path.

    Building the reply got 7.2x to 9.0x faster than on the base for those five mainnet blocks and the two largest regtest blocks (median of three runs, Release, Apple M5). Peak heap fell from 2.9 to 2.0 MB for 481824 and from 7.9 to 4.0 MB for 774628. I measured with a Boost test that writes each block with WriteBlock and reads it back both ways, and a functional test running a head and a base node; I can share both as a gist.

  20. DrahtBot added the label Needs rebase on Oct 28, 2026
  21. DrahtBot commented at 11:18 PM on October 28, 2026: contributor

    <!--cf906140f33d8803c4a75a2196329ecb-->

    🐙 This pull request conflicts with the target branch and needs rebase.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-10-30 09:40 UTC