init: fix reindex deadlock by waking cv after interrupt #35652

pull ismaelsadeeq wants to merge 1 commits into bitcoin:master from ismaelsadeeq:07-2026-fix-init-genesis-deadlock changing 2 files +14 −3
  1. ismaelsadeeq commented at 4:27 PM on July 3, 2026: member

    During startup with -reindex, the block index and chainstate are wiped, and the normal LoadGenesisBlock() startup path is skipped while block files are being indexed. The init thread can then wait for genesis to be processed via the tip-block condition variable.

    If shutdown is requested after the init thread enters that wait but before the import thread activates genesis, the reindex scan can return early without calling the later LoadGenesisBlock() fallback or ActivateBestChains(). Because no block tip is connected, the block-tip notification never fires. The wait predicate would accept ShutdownRequested(node), but it is not evaluated again unless the condition variable is notified.

    Notify the tip-block condition variable after ImportBlocks() returns so the genesis wait can observe shutdown and exit cleanly.

    Described in detail #35621 (comment)

    The current test covers the path, but won't fail deterministically. You can use L0rinc suggested patch for a deterministic patch that demonstrate the deadlock on master #35652#pullrequestreview-4627988553

  2. DrahtBot commented at 4:27 PM on July 3, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    Code Coverage & Benchmarks

    For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35652.

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK maflcko, mzumsande, sedited
    Stale ACK l0rinc

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #35071 (Reindex: save progress to continue after interruption by pinheadmz)
    • #33854 (fix assumevalid is ignored during reindex by Eunovo)
    • #29700 (kernel, refactor: return error status on all fatal errors by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  3. in src/init.cpp:2053 in 4c72802993
    2048 | @@ -2049,6 +2049,12 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
    2049 |          ScheduleBatchPriority();
    2050 |          // Import blocks and ActivateBestChain()
    2051 |          ImportBlocks(chainman, vImportFiles);
    2052 | +        // An interrupted import may return without activating genesis. Wake
    2053 | +        // the "Wait for genesis block to be processed" wait, which is
    


    l0rinc commented at 8:49 PM on July 3, 2026:

    "Wait for genesis block to be processed" wait

    nit: sound a bit awkward, could we avoid the repeated "wait" wording here?

            // the init thread's genesis wait, which is
    
  4. in test/functional/feature_reindex.py:28 in 4c72802993
      24 |  from test_framework.util import (
      25 |      assert_equal,
      26 |      util_xor,
      27 |  )
      28 |  
      29 | +UNKNOWN_ENTRIES = 2_000_000
    


    l0rinc commented at 9:34 PM on July 3, 2026:

    This test is already one of the heavier functional tests, see #35576. 2_000_000 entries writes almost 200 MB of synthetic block data before the scan starts.

    Could we lower the disk and memory cost a bit? If that turns out flaky, we could keep the default test cheap and put the heavier variant in the EXTENDED_SCRIPTS instead.

    UNKNOWN_ENTRIES = 500_000
    
  5. in test/functional/feature_reindex.py:152 in 4c72802993
     147 |          self.reindex(False)
     148 |          self.reindex(True)
     149 |  
     150 |          self.out_of_order()
     151 | +        self.shutdown_during_reindex_before_genesis()
     152 |          self.continue_reindex_after_shutdown()
    


    l0rinc commented at 9:53 PM on July 3, 2026:

    If we also keep the earlier suggestion that preserves the original blk00000.dat, the shutdown-before-genesis scenario makes that block file much larger. Running continue_reindex_after_shutdown() after it means the later subtest has to scan the larger file too.

    <details><summary>`continue_reindex_after_shutdown()` before `shutdown_during_reindex_before_genesis()`</summary>

    diff --git a/test/functional/feature_reindex.py b/test/functional/feature_reindex.py
    --- a/test/functional/feature_reindex.py	(revision 00927b93b4a27a231a7c3f3e038f22e528507dc6)
    +++ b/test/functional/feature_reindex.py	(revision 621f11d2c2f2ccdda3102f14837a4d3e7c031b3e)
    @@ -111,6 +111,8 @@
         def shutdown_during_reindex_before_genesis(self):
             """Interrupt -reindex before genesis is activated."""
             node = self.nodes[0]
    +        if not node.running:
    +            self.start_node(0)
             blockcount = node.getblockcount()
             self.stop_node(0)
             self.log.info(f"Prepend {UNKNOWN_ENTRIES} genesis-free out-of-order entries to blk00000.dat")
    @@ -152,8 +154,8 @@
             self.reindex(True)
     
             self.out_of_order()
    +        self.continue_reindex_after_shutdown()
             self.shutdown_during_reindex_before_genesis()
    -        self.continue_reindex_after_shutdown()
     
     
     if __name__ == '__main__':
    

    </details>

  6. in test/functional/feature_reindex.py:142 in 4c72802993
     137 | +        except AssertionError:
     138 | +            node.kill_process()
     139 | +            raise AssertionError("node hung in init's genesis wait after shutdown was requested")
     140 | +        self.log.info("Check that a restart resumes the reindex and recovers genesis")
     141 | +        self.start_node(0)
     142 | +        assert_equal(node.getblockcount(), 0)
    


    l0rinc commented at 10:14 PM on July 3, 2026:

    The restart check is currently weak, there are no generated blocks left to reindex, so getblockcount() == 0 only proves genesis is available again.

    Could we keep the original block data and assert that restart returns to the height recorded before the interrupt? That would cheaply extend the test from "the node exits and starts again" to "the node can resume reindexing through the existing blocks after restart."

    <details><summary>check full reindex recovery</summary>

    diff --git a/test/functional/feature_reindex.py b/test/functional/feature_reindex.py
    --- a/test/functional/feature_reindex.py	(revision a0bb023899acb77176dad188f5698c4ea026f915)
    +++ b/test/functional/feature_reindex.py	(revision 0d1b26f1e8bad56d0ac8b4307331ea1014d76f69)
    @@ -111,15 +111,19 @@
         def shutdown_during_reindex_before_genesis(self):
             """Interrupt -reindex before genesis is activated."""
             node = self.nodes[0]
    +        blockcount = node.getblockcount()
             self.stop_node(0)
    -        self.log.info(f"Replace blk00000.dat with {UNKNOWN_ENTRIES} genesis-free out-of-order entries")
    +        self.log.info(f"Prepend {UNKNOWN_ENTRIES} genesis-free out-of-order entries to blk00000.dat")
             # Repeated header-only records with unknown parents keep the import
    -        # thread scanning before genesis can be activated.
    +        # thread scanning before genesis can be activated. The entry size is a
    +        # multiple of the XOR key size, so the shifted original block file
    +        # content remains correctly XORed after the prefix.
             header = CBlockHeader().serialize()
             entry = MAGIC_BYTES["regtest"] + len(header).to_bytes(4, "little") + header
             assert_equal(len(entry) % len(node.read_xor_key()), 0)
             entry = util_xor(entry, node.read_xor_key(), offset=0)
    -        (node.blocks_path / "blk00000.dat").write_bytes(entry * UNKNOWN_ENTRIES)
    +        block_file = node.blocks_path / "blk00000.dat"
    +        block_file.write_bytes(entry * UNKNOWN_ENTRIES + block_file.read_bytes())
             self.log.info("Request shutdown while the reindex scan is before genesis activation")
             with node.assert_debug_log(expected_msgs=["Interrupt requested. Exit reindexing."],
                                        unexpected_msgs=["UpdateTip"], timeout=60):
    @@ -139,7 +143,7 @@
                 raise AssertionError("node hung in init's genesis wait after shutdown was requested")
             self.log.info("Check that a restart resumes the reindex and recovers genesis")
             self.start_node(0)
    -        assert_equal(node.getblockcount(), 0)
    +        assert_equal(node.getblockcount(), blockcount)
     
         def run_test(self):
             self.reindex(False)
    

    </details>

  7. l0rinc approved
  8. l0rinc commented at 10:28 PM on July 3, 2026: contributor

    ACK 4c728029933e0c69b813745f9fea72fdc2456aa0

    Reviewed the change and reproduced the issue locally with a temporary monkey patch that requests interrupt before loading blk00000.dat during -reindex.

    <details><summary>Local monkey-patch reproducer</summary>

    diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
    --- a/src/node/blockstorage.cpp	(revision 1835f2fcbf482035e1c11c72ae39cee4e24e4c00)
    +++ b/src/node/blockstorage.cpp	(revision 5d358314134771fb2b344f966b8cab2440ede334)
    @@ -43,6 +43,7 @@
    
     #include <cerrno>
     #include <compare>
    +#include <cstdlib>
     #include <cstddef>
     #include <cstdio>
     #include <exception>
    @@ -1295,6 +1296,12 @@
                     break; // This error is logged in OpenBlockFile
                 }
                 LogInfo("Reindexing block file blk%05u.dat (%d%% complete)...", (unsigned int)nFile, nFile * 100 / total_files);
    +            if (std::getenv("WAIT_FOR_INTERRUPT_BEFORE_GENESIS")) {
    +                LogInfo("Monkey patch: setting interrupt before loading blk%05u.dat", (unsigned int)nFile);
    +                if (!const_cast<util::SignalInterrupt&>(chainman.m_interrupt)()) LogError("Monkey patch: failed to set interrupt");
    +                LogInfo("Interrupt requested. Exit reindexing.");
    +                return;
    +            }
                 chainman.LoadExternalBlockFile(file, &pos, &blocks_with_unknown_parent);
                 if (chainman.m_interrupt) {
                     LogInfo("Interrupt requested. Exit reindexing.");
    

    Exercised before/after with:

    DATADIR="$(mktemp -d)"
    cmake --build build -j 8 --target bitcoind
    build/bin/bitcoind -datadir="$DATADIR" -stopatheight=100
    WAIT_FOR_INTERRUPT_BEFORE_GENESIS=1 build/bin/bitcoind -datadir="$DATADIR" -reindex
    

    Before the fix, bitcoind -reindex logged Interrupt requested. Exit reindexing. and initload thread exit, but stayed alive until I interrupted it manually. With this change, the same reproducer exits on its own and reaches Shutdown done.

    </details>

    I left a few non-blocking test suggestions around test cost and making the restart check cover reindex recovery, but nothing blocking. Someone else familiar with the code should also review it.

  9. in test/functional/feature_reindex.py:132 in 4c72802993
     127 | +                node.start(
     128 | +                    extra_args=["-reindex"],
     129 | +                    **({"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP} if platform.system() == "Windows" else {}),
     130 | +                )
     131 | +            with node.busy_wait_for_debug_log([b"initload thread exit"]):
     132 | +                time.sleep(0.1)
    


    maflcko commented at 8:51 AM on July 6, 2026:

    not sure about this test. Seems fine to let reviewers know it exists, but the value of having to maintain this seems not worth it.

    Overall, it seems racy (with various sleeps and large magic values), so while it may work today in the current CI, there is no expectation that it will continue to work in the future without additional work.

    Going forward, I wonder if a better fix would be to simply require that the first block in the first block file is the genesis block.

    The question is, whether there is any version or use-case out there that requires support for a missing genesis block as the first block in the first file during -reindex? I'd say no, because the other code doesn't assume so. At least, I presume it will break the // Activate the genesis block so normal node progress can continue feature: If the genesis block was never present or only present in the end, the node would wait in init for the whole duration of the -reindex.

    For reference, one could come up with an alternative fix like this:

    diff --git a/src/node/blockstorage.cpp b/src/node/blockstorage.cpp
    index 8726c741fa..2967802b6a 100644
    --- a/src/node/blockstorage.cpp
    +++ b/src/node/blockstorage.cpp
    @@ -1278,4 +1278,6 @@ void ImportBlocks(ChainstateManager& chainman, std::span<const fs::path> import_
         ImportingNow imp{chainman.m_blockman.m_importing};
     
    +    // Note that the genesis block is already written/loaded here.
    +
         // -reindex
         if (!chainman.m_blockman.m_blockfiles_indexed) {
    @@ -1305,6 +1307,4 @@ void ImportBlocks(ChainstateManager& chainman, std::span<const fs::path> import_
             chainman.m_blockman.m_blockfiles_indexed = true;
             LogInfo("Reindexing finished");
    -        // To avoid ending up in a situation without genesis block, re-try initializing (no-op if reindexing worked):
    -        chainman.ActiveChainstate().LoadGenesisBlock();
         }
     
    diff --git a/src/node/chainstate.cpp b/src/node/chainstate.cpp
    index 1725fe70bd..e46edf0eb4 100644
    --- a/src/node/chainstate.cpp
    +++ b/src/node/chainstate.cpp
    @@ -60,8 +60,7 @@ static ChainstateLoadResult CompleteChainstateInitialization(
     
         // At this point blocktree args are consistent with what's on disk.
    -    // If we're not mid-reindex (based on disk + args), add a genesis block on disk
    -    // (otherwise we use the one already on disk).
    -    // This is called again in ImportBlocks after the reindex completes.
    -    if (chainman.m_blockman.m_blockfiles_indexed && !chainman.ActiveChainstate().LoadGenesisBlock()) {
    +    // Even in an ongoing -reindex (based on disk + args), force-write a
    +    // genesis block if missing.
    +    if (!chainman.ActiveChainstate().LoadGenesisBlock()) {
             return {ChainstateLoadStatus::FAILURE, _("Error initializing block database")};
         }
    

    However, that breaks feature_reindex_readonly.py. So the fix would probably have to only write the genesis block if it isn't already present.

    I mention it, because it seems easier to assume the genesis block always exists and is active as soon as init starts. (See also #34440#issue-3869652089: "Further ideas: Change CChain to always have at least one element (genesis), that way there is always genesis and tip.")


    ismaelsadeeq commented at 10:07 AM on July 6, 2026:

    I will limit this to waking up the cv. Your suggestion is okay, but I'm also fine with the genesis not at the beginning. Changing the assumption is beyond the scope of this change.


    maflcko commented at 10:46 AM on July 6, 2026:

    I also wonder if there could be a smaller test:

    • Maybe a small addition to ./test/functional/feature_init.py to add a new reindex log line and append a -reindex when it is present?
    • I presume the test would never fail on master, but maybe a manual patch that injects a sleep in the right place can be provided for reviewers?

    ismaelsadeeq commented at 12:00 PM on July 6, 2026:

    I like this suggestion 🚀 . I will push this, thanks.


    ismaelsadeeq commented at 12:51 PM on July 6, 2026:

    I presume the test would never fail on master

    Surprisingly, it did fail a lot for me, just not deterministically, I will link the reproducer in the PR description.

  10. ismaelsadeeq force-pushed on Jul 6, 2026
  11. ismaelsadeeq commented at 10:05 AM on July 6, 2026: member

    Forced pushed from 4c728029933e0c69b813745f9fea72fdc2456aa0 to 85de9416ff2b959e34b65509a8c710849de08478 4c72802993...85de9416ff

  12. in src/init.cpp:2057 in 85de9416ff
    2048 | @@ -2049,6 +2049,12 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
    2049 |          ScheduleBatchPriority();
    2050 |          // Import blocks and ActivateBestChain()
    2051 |          ImportBlocks(chainman, vImportFiles);
    2052 | +        // An interrupted import may return without activating genesis. Wake
    2053 | +        // the init thread's genesis wait, which is otherwise only notified
    2054 | +        // on blockTip, and that never fires when the import was interrupted
    2055 | +        // before activating genesis. This wakeup lets the wait observe the
    2056 | +        // shutdown request.
    2057 | +        WITH_LOCK(node.notifications->m_tip_block_mutex, node.notifications->m_tip_block_cv.notify_all());
    


    maflcko commented at 10:40 AM on July 6, 2026:

    Could use chainman.GetNotifications() to avoid having to think about nullptr deref?


    ismaelsadeeq commented at 11:41 AM on July 6, 2026:

    chainman.GetNotifications() returns kernel::Notifications&, the abstract notification interface. The interface does not expose m_tip_block_mutex and m_tip_block_cv.

    But I can capture the asserted kernel_notifcations variable and use it here?

  13. maflcko approved
  14. maflcko commented at 10:46 AM on July 6, 2026: member

    lgtm

  15. init: wake genesis wait after ImportBlocks() returns
    If shutdown interrupts ImportBlocks() before genesis activation,
    no blockTip notification is sent. The wait predicate allows
    shutdown to occur, but the condition variable is never
    notified, so init can remain stuck waiting for genesis activation.
    
    Notify the tip condition variable after ImportBlocks() returns so
    interrupted imports wake the wait and let it observe the shutdown request.
    
    Add test coverage for interrupting startup after reindex block files import
    begins, which exercises the path where ImportBlocks() can return before genesis
    activation.
    c1313b199f
  16. ismaelsadeeq force-pushed on Jul 6, 2026
  17. ismaelsadeeq commented at 12:45 PM on July 6, 2026: member

    Forced pushed 85de9416ff...c1313b199f

    • I added a minimal test that could catch the issue but not deterministically. The test can be caught with my functional test here #35621 (comment) or using @l0rinc review here #35652#pullrequestreview-4627988553 thanks 👍🏾

    • #35652 (review) done thanks @maflcko

  18. maflcko commented at 1:30 PM on July 6, 2026: member

    Thanks for including the minimal test. For me, it fails on the first attempt without recompiling, and passes as soon as I re-compile.

    review ACK c1313b199fe320c8d3e7ba01577378dfb85009cf 🦁

    <details><summary>Show signature</summary>

    Signature:

    untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    trusted comment: review ACK c1313b199fe320c8d3e7ba01577378dfb85009cf 🦁
    jLs9y97CfzvSqs0Ufjje3qO8di3LpJhn9i+BlaJVI3tuJ7KqDqWigH5YjG/psY7RZnxYq6mAMgZBUbpjBsQJCw==
    

    </details>

  19. DrahtBot requested review from l0rinc on Jul 6, 2026
  20. in src/init.cpp:2053 in c1313b199f
    2049 | +    node.background_init_thread = std::thread(&util::TraceThread, "initload", [=, &chainman, &args, &kernel_notifications, &node] {
    2050 |          ScheduleBatchPriority();
    2051 |          // Import blocks and ActivateBestChain()
    2052 |          ImportBlocks(chainman, vImportFiles);
    2053 | +        // An interrupted import may return without activating genesis. Wake
    2054 | +        // the init thread's genesis wait, which is otherwise only notified
    


    mzumsande commented at 1:40 PM on July 6, 2026:

    Just noting that there is also a m_tip_block_cv notification in Interrupt(), but that would run from the same init thread that is being blocked and is thus never executed in the interrupted reindex case.

  21. mzumsande commented at 1:48 PM on July 6, 2026: contributor

    Code Review ACK c1313b1

  22. sedited approved
  23. sedited commented at 1:52 PM on July 6, 2026: contributor

    tACK c1313b199fe320c8d3e7ba01577378dfb85009cf

  24. sedited merged this on Jul 6, 2026
  25. sedited closed this on Jul 6, 2026

  26. ismaelsadeeq deleted the branch on Jul 6, 2026
  27. l0rinc commented at 5:44 PM on July 6, 2026: contributor

    post-merge code review ACK c1313b199fe320c8d3e7ba01577378dfb85009cf


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-07-09 06:47 UTC