txospenderindex: disable bloom filters to optimize disk usage #35568

pull andrewtoth wants to merge 2 commits into bitcoin:master from andrewtoth:txospenderindex-disable-bloom changing 7 files +14 −11
  1. andrewtoth commented at 2:34 PM on June 19, 2026: contributor

    LevelDB bloom filters are only consulted on Get point reads. This can be verified in https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/table/table.cc#L224-L228. InternalGet is the only place that consults the filter, and it is only reached via a Get or Exists point read. The filters are never consulted for iterator seeks with an iterator created via NewIterator. txospenderindex only reads via iterator seeks, so building them is wasted effort and space.

    For a db as large as txospenderindex, this results in measurable performance and disk usage. On master, a full sync took 2h46m, and the resulting db was 94.5 GB. On this branch, a full sync took 2h32m, and the resulting db was 90 GB. So this is a sync speedup of 14 minutes, and a disk space reduction of 4.5 GB.

  2. DrahtBot commented at 2:34 PM on June 19, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35568.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK sedited, l0rinc

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35620 (leveldb: move unused block-cache budget to write buffers by l0rinc)
    • #35531 (txindex: hash keys and pack positions to reduce disk usage by andrewtoth)
    • #34132 (coins: drop error catcher, centralize fatal read handling by l0rinc)
    • #33324 (blocks: add -reobfuscate-blocks argument to enable (de)obfuscating existing blocks by l0rinc)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. andrewtoth force-pushed on Jun 19, 2026
  4. sedited approved
  5. sedited commented at 7:12 PM on June 19, 2026: contributor

    Nice find, ACK 74dfc8ec49c7bb480dac0c785cc2aae1a5e3ca30

  6. in src/index/base.cpp:75 in 74dfc8ec49 outdated
      71 |          .path = path,
      72 |          .cache_bytes = n_cache_size,
      73 |          .memory_only = f_memory,
      74 |          .wipe_data = f_wipe,
      75 |          .obfuscate = f_obfuscate,
      76 | +        .bloom_filter = f_bloom,
    


    l0rinc commented at 5:39 PM on June 21, 2026:

    Could we add a simple unit test to validate this new use case, since no existing dbwrapper_tests case constructs that configuration explicitly.

    We could add a small wrapper round trip that writes, reads, checks existence, and iterates with .bloom_filter = false so the option path stays covered independently of txospenderindex sync behavior:

    BOOST_AUTO_TEST_CASE(dbwrapper_no_bloom_filter)
    {
        fs::path ph = m_args.GetDataDirBase() / "dbwrapper_no_bloom_filter";
        CDBWrapper dbw({.path = ph, .cache_bytes = 1_MiB, .wipe_data = true, .bloom_filter = false});
    
        const uint8_t key{'a'};
        const uint8_t key2{'b'};
        const uint256 value{m_rng.rand256()};
        const uint256 value2{m_rng.rand256()};
    
        dbw.Write(key, value);
        dbw.Write(key2, value2);
        BOOST_CHECK(dbw.Exists(key));
    
        uint256 read_value;
        BOOST_REQUIRE(dbw.Read(key, read_value));
        BOOST_CHECK_EQUAL(read_value, value);
    
        std::unique_ptr<CDBIterator> it(dbw.NewIterator());
        it->Seek(key);
    
        uint8_t read_key;
        BOOST_REQUIRE(it->Valid());
        BOOST_REQUIRE(it->GetKey(read_key));
        BOOST_CHECK_EQUAL(read_key, key);
        BOOST_REQUIRE(it->GetValue(read_value));
        BOOST_CHECK_EQUAL(read_value, value);
    
        it->Next();
        BOOST_REQUIRE(it->Valid());
        BOOST_REQUIRE(it->GetKey(read_key));
        BOOST_CHECK_EQUAL(read_key, key2);
        BOOST_REQUIRE(it->GetValue(read_value));
        BOOST_CHECK_EQUAL(read_value, value2);
    
        it->Next();
        BOOST_CHECK(!it->Valid());
    }
    

    Alternatively maybe we could fuzz this to have a random value.


    andrewtoth commented at 1:52 AM on June 23, 2026:

    Not sure a unit test like that is useful. It doesn't tell us if the bloom filter was used or not. I added the option to the fuzz harness though.


    l0rinc commented at 5:58 PM on June 23, 2026:

    It doesn't tell us if the bloom filter was used or not

    Agree, but it tells us if we can still read with & without bloom filters. Maybe a loop both with and without this option (doing the same) would be reassuring.


    andrewtoth commented at 5:54 PM on June 24, 2026:

    Isn't that already well covered with existing unit and functional tests? And it will also now be covered by fuzz testing.


    l0rinc commented at 6:24 PM on June 24, 2026:

    Please correct me if I misunderstood something, but before this change CDBWrapper always configured LevelDB with NewBloomFilterPolicy(10), so we never explicitly exercised filter_policy == nullptr.

    I agree that a standalone test (like the one I posted above) may be too verbose, but maybe we can extend the existing obfuscation matrix - and add CompactFull() so the read path exercises SST files created with and without a filter policy:

    diff --git a/src/test/dbwrapper_tests.cpp b/src/test/dbwrapper_tests.cpp
    --- a/src/test/dbwrapper_tests.cpp	(revision fac744efa7745f4d02c4e17a0c5fdd6da3d92ae4)
    +++ b/src/test/dbwrapper_tests.cpp	(date 1782325217634)
    @@ -22,7 +22,7 @@
     BOOST_AUTO_TEST_CASE(dbwrapper)
     {
         // Perform tests both obfuscated and non-obfuscated.
    -    for (const bool obfuscate : {false, true}) {
    +    for (const auto [obfuscate, bloom] : {std::pair{true, true}, std::pair{true, false} std::pair{false, true}, std::pair{false, false}}) {
             constexpr size_t CACHE_SIZE{1_MiB};
             const fs::path path{m_args.GetDataDirBase() / "dbwrapper"};
     
    @@ -31,7 +31,7 @@
     
             // Write values
             {
    -            CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .wipe_data = true, .obfuscate = obfuscate}};
    +            CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .wipe_data = true, .obfuscate = obfuscate, .bloom_filter = bloom}};
                 BOOST_CHECK_EQUAL(obfuscate, !dbw.IsEmpty());
     
                 // Ensure that we're doing real obfuscation when obfuscate=true
    @@ -44,17 +44,18 @@
                     dbw.Write(key, value);
                     key_values.emplace_back(key, value);
                 }
    +            dbw.CompactFull(); // Force SST creation so reads exercise the filter policy
             }
     
             // Verify that the obfuscation key is never obfuscated
             {
    -            CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .obfuscate = false}};
    +            CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .obfuscate = false, .bloom_filter = bloom}};
                 BOOST_CHECK_EQUAL(obfuscation, dbwrapper_private::GetObfuscation(dbw));
             }
     
             // Read back the values
             {
    -            CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .obfuscate = obfuscate}};
    +            CDBWrapper dbw{{.path = path, .cache_bytes = CACHE_SIZE, .obfuscate = obfuscate, .bloom_filter = bloom}};
     
                 // Ensure obfuscation is read back correctly
                 BOOST_CHECK_EQUAL(obfuscation, dbwrapper_private::GetObfuscation(dbw));
    @@ -62,10 +63,12 @@
     
                 // Verify all written values
                 for (const auto& [key, expected_value] : key_values) {
    +                BOOST_CHECK(dbw.Exists(key));
                     uint256 read_value{};
                     BOOST_CHECK(dbw.Read(key, read_value));
                     BOOST_CHECK_EQUAL(read_value, expected_value);
                 }
    +            BOOST_CHECK(!dbw.Exists(uint8_t{11}));
             }
         }
     }
    @@ -197,9 +200,9 @@
     BOOST_AUTO_TEST_CASE(dbwrapper_iterator)
     {
         // Perform tests both obfuscated and non-obfuscated.
    -    for (const bool obfuscate : {false, true}) {
    +    for (const auto [obfuscate, bloom] : {std::pair{true, true}, std::pair{true, false}, std::pair{false, true}, std::pair{false, false}}) {
             fs::path ph = m_args.GetDataDirBase() / (obfuscate ? "dbwrapper_iterator_obfuscate_true" : "dbwrapper_iterator_obfuscate_false");
    -        CDBWrapper dbw({.path = ph, .cache_bytes = 1_MiB, .memory_only = true, .wipe_data = false, .obfuscate = obfuscate});
    +        CDBWrapper dbw({.path = ph, .cache_bytes = 1_MiB, .memory_only = true, .wipe_data = false, .obfuscate = obfuscate, .bloom_filter = bloom});
     
             // The two keys are intentionally chosen for ordering
             uint8_t key{'j'};
    

    andrewtoth commented at 7:54 PM on June 24, 2026:

    This option goes right to leveldb, it's not really used at all in the wrapper aside from creating the filters or not. Creating the filters or not is tested in the leveldb test suite.

    Whether reads work correctly without the filters is tested by existing unit and functional tests.

    I don't think more tests are adding any value here.


    l0rinc commented at 7:57 PM on June 24, 2026:

    k, thanks for considering, please resolve the thread - I already ACKed :)

  7. l0rinc approved
  8. l0rinc commented at 5:43 PM on June 21, 2026: contributor

    Concept ACK, I'll test it myself and get back to you. What happens when the user has an existing index, do they need to wait for compaction? Should we extend force_compact for these? Should we add a release notes for this?

  9. andrewtoth force-pushed on Jun 23, 2026
  10. andrewtoth commented at 1:56 AM on June 23, 2026: contributor

    What happens when the user has an existing index, do they need to wait for compaction?

    Yes, size compactions will remove the bloom filters as files are replaced.

    Should we extend force_compact for these?

    I believe starting up with -forcecompactdb=1 also fully compacts the index dbs, so this is covered already.

    Should we add a release notes for this?

    I'm not sure an explicit release note is warranted. This could be covered under This release includes new features, various bug fixes and performance improvements, as well as updated translations..

  11. l0rinc commented at 4:13 AM on June 24, 2026: contributor

    tested ACK b343c17ee06c6357b564b05177f6060c4da81e70

    I can reproduce the results, ~19% faster, ~5% smaller. ~I'm still running tests to see if forcecompact does indeed work with this~ Edit: forcecompact also seems to work correctly.

    <details><summary>benchmark</summary>

    BEFORE="794befd4b06dcace1adbdb15e65bba1667f9bc15"; AFTER="74dfc8ec49c7bb480dac0c785cc2aae1a5e3ca30"; \                                                                                                                                                                                           
    DATA_DIR="/mnt/my_storage/BitcoinData"; export DATA_DIR; \                                                                                                                                                                                                                                                                                
    LOG="${DATA_DIR}/debug.log"; export LOG; \                                                                                                                                                                                                                                                                                                
    SIZE_LOG="${PWD}/txospenderindex-size.log"; export SIZE_LOG; : > "${SIZE_LOG}"; \                                                                                                                                                                                                                                                         wait_index() { pid="$1"; while kill -0 "$pid" 2>/dev/null; do grep -q 'txospenderindex is enabled at height' "${LOG}" 2>/dev/null && return 0; sleep 5; done; grep -q 'txospenderindex is enabled at height' "${LOG}" 2>/dev/null; }; export -f wait_index; \                                                                             
    run_index() { label="$1"; daemon="$2"; shift 2; cli="${daemon%bitcoind}bitcoin-cli"; "$daemon" "$@" & pid=$!; if ! wait_index "$pid"; then wait "$pid" || true; echo "$label: bitcoind exited before txospenderindex was enabled" >&2; tail -100 "${LOG}" >&2 || true; return 1; fi; "$cli" -datadir="${DATA_DIR}" stop >/dev/null || kill "$pid" 2>/dev/null || true; wait "$pid" || true; sleep 10; du -sh "${DATA_DIR}/indexes/txospenderindex" | awk -v label="$label" '{print label " size: " $1}' >> "${SIZE_LOG}"; du -sb "${DATA_DIR}/indexes/txospenderindex" | awk -v label="$label" '{print label " bytes: " $1}' >> "${SIZE_LOG}"; }; export -f run_index; \
    git reset --hard >/dev/null 2>&1 && git clean -fxd >/dev/null 2>&1 && git fetch origin $BEFORE $AFTER >/dev/null 2>&1; \                                             
    for c in $BEFORE:build-before $AFTER:build-after; do \                            
      git checkout ${c%:*} >/dev/null 2>&1 && cmake -B ${c#*:} -G Ninja -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1 && ninja -C ${c#*:} bitcoind bitcoin-cli >/dev/null 2>&1; \                                                                                                                                                                
    done; \                                                                           
    echo "txospenderindex bloom filter | $(hostname) | $(uname -m) | $(lscpu | grep 'Model name' | head -1 | cut -d: -f2 | xargs) | $(nproc) threads | $(free -h | awk '/^Mem:/{print $2}') RAM | $(df -T $DATA_DIR | awk 'NR==2{print $2}') | $(lsblk -no ROTA $(df --output=source $DATA_DIR | tail -1) | grep -q 1 && echo HDD || echo SSD)" && \                                                                            
    hyperfine --runs 1 --shell bash --sort command \                                  
      --prepare "rm -rf ${DATA_DIR}/indexes/* ${DATA_DIR}/debug.log" \                
      "run_index before ./build-before/bin/bitcoind -datadir=${DATA_DIR} -txospenderindex=1 -connect=0 -printtoconsole=0" \                                              
      "run_index after  ./build-after/bin/bitcoind  -datadir=${DATA_DIR} -txospenderindex=1 -connect=0 -printtoconsole=0"; \                                             
    echo "on-disk sizes:" && cat "${SIZE_LOG}"      
    

    </details>

    txospenderindex bloom filter | ssd-ryzen | x86_64 | AMD Ryzen 7 3700X 8-Core Processor | 16 threads | 62Gi RAM | ext4 | SSD

    Benchmark 1: run_index before ./build-before/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -txospenderindex=1 -connect=0 -printtoconsole=0                                                                   
      Time (abs ≡):        16854.323 s               [User: 18140.234 s, System: 2824.711 s]                                                                             
    
    Benchmark 2: run_index after  ./build-after/bin/bitcoind  -datadir=/mnt/my_storage/BitcoinData -txospenderindex=1 -connect=0 -printtoconsole=0                               
      Time (abs ≡):        14187.279 s               [User: 15187.570 s, System: 2748.195 s]                                                                             
    
    Relative speed comparison                                                         
            1.19          run_index before ./build-before/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -txospenderindex=1 -connect=0 -printtoconsole=0
            1.00          run_index after  ./build-after/bin/bitcoind  -datadir=/mnt/my_storage/BitcoinData -txospenderindex=1 -connect=0 -printtoconsole=0
    

    on-disk sizes:
    before size: 88G
    before bytes: 94306922005
    after size: 84G
    after bytes: 89883517156

    <details><summary>forcecompact</summary>

    BEFORE="794befd4b06dcace1adbdb15e65bba1667f9bc15"; AFTER="74dfc8ec49c7bb480dac0c785cc2aae1a5e3ca30"; \ 
    DATA_DIR="/mnt/my_storage/BitcoinData"; export DATA_DIR; \
    LOG="${DATA_DIR}/debug.log"; export LOG; \
    SIZE_LOG="${PWD}/txospenderindex-forcecompact-size.log"; export SIZE_LOG; : > "${SIZE_LOG}"; \
    wait_log() { pattern="$1"; pid="$2"; while kill -0 "$pid" 2>/dev/null; do grep -Fq "$pattern" "${LOG}" 2>/dev/null && return 0; sleep 10; done; grep -Fq "$pattern" "${LOG}" 2>/dev/null; }; export -f wait_log; \
    stop_node() { pid="$1"; cli="$2"; "$cli" -datadir="${DATA_DIR}" stop >/dev/null 2>&1 || kill "$pid" 2>/dev/null || true; wait "$pid" || true; sleep 10; }; export -f stop_node; \
    record_size() { label="$1"; du -sh "${DATA_DIR}/indexes/txospenderindex" | awk -v label="$label" '{print label " size: " $1}' >> "${SIZE_LOG}"; du -sb "${DATA_DIR}/indexes/txospenderindex" | awk -v label="$label" '{print label " bytes: " $1}' >> "${SIZE_LOG}"; }; export -f record_size; \
    git reset --hard >/dev/null 2>&1 && git clean -fxd >/dev/null 2>&1 && git fetch origin $BEFORE $AFTER >/dev/null 2>&1; \
    for c in $BEFORE:build-before $AFTER:build-after; do \
      git checkout ${c%:*} >/dev/null 2>&1 && cmake -B ${c#*:} -G Ninja -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1 && ninja -C ${c#*:} bitcoind bitcoin-cli >/dev/null 2>&1; \
    done; \
    echo "recreate old txospenderindex, then forcecompact new no-bloom txospenderindex"; \
    rm -rf "${DATA_DIR}/indexes/txospenderindex" "${DATA_DIR}/debug.log"; \
    ./build-before/bin/bitcoind -datadir="${DATA_DIR}" -txospenderindex=1 -connect=0 -printtoconsole=0 & pid=$!; \
    wait_log "txospenderindex is enabled at height" "$pid" && stop_node "$pid" "./build-before/bin/bitcoin-cli" && record_size "before old-rebuilt"; \
    : > "${DATA_DIR}/debug.log"; \
    ./build-after/bin/bitcoind -datadir="${DATA_DIR}" -txospenderindex=1 -forcecompactdb=1 -connect=0 -printtoconsole=0 & pid=$!; \
    wait_log "Finished database compaction of ${DATA_DIR}/indexes/txospenderindex/db" "$pid" && wait_log "txospenderindex is enabled at height" "$pid" && stop_node "$pid" "./build-after/bin/bitcoin-cli" && record_size "after new-forcecompact"; \
    echo "on-disk sizes:" && cat "${SIZE_LOG}"
    recreate old txospenderindex, then forcecompact new no-bloom txospenderindex
    [1] 1435394
    [1]+  Done                    ./build-before/bin/bitcoind -datadir="${DATA_DIR}" -txospenderindex=1 -connect=0 -printtoconsole=0
    [1] 1451378
    [1]+  Done                    ./build-after/bin/bitcoind -datadir="${DATA_DIR}" -txospenderindex=1 -forcecompactdb=1 -connect=0 -printtoconsole=0
    

    </details>

    on-disk sizes: before old-rebuilt size: 88G before old-rebuilt bytes: 94381614322 after new-forcecompact size: 84G after new-forcecompact bytes: 89519621524

  12. DrahtBot requested review from sedited on Jun 24, 2026
  13. l0rinc approved
  14. sedited approved
  15. sedited commented at 4:20 PM on June 29, 2026: contributor

    Re-ACK b343c17ee06c6357b564b05177f6060c4da81e70

  16. l0rinc commented at 5:06 PM on July 1, 2026: contributor

    Added #35634 to complement this change

  17. sedited requested review from fjahr on Jul 1, 2026
  18. sedited referenced this in commit b393985aa0 on Jul 4, 2026
  19. sedited commented at 8:48 AM on July 4, 2026: contributor

    Can you add this to #35634's release note?

  20. txospenderindex: disable bloom filters to optimize disk usage
    LevelDB bloom filters are only consulted on Get point reads.
    They are not consulted for iterator seeks.
    txospenderindex only reads via iterator seeks, so building
    them is wasted effort and space.
    a2b1c86903
  21. doc: add release notes 6d0ea4cf5b
  22. andrewtoth force-pushed on Jul 4, 2026
  23. andrewtoth commented at 3:33 PM on July 4, 2026: contributor

    Rebased for #35634. Generalized the release note so it will apply to both changes.

  24. sedited approved
  25. sedited commented at 3:34 PM on July 4, 2026: contributor

    Re-ACK 6d0ea4cf5bdc97331d4550a35d361a7d13d259a4

  26. DrahtBot requested review from l0rinc on Jul 4, 2026
  27. l0rinc commented at 3:41 PM on July 4, 2026: contributor

    ACK 6d0ea4cf5bdc97331d4550a35d361a7d13d259a4

    (the PR description needs updated numbers now)


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-07-09 06:47 UTC