validation: return block-file flush errors #35714

pull l0rinc wants to merge 2 commits into bitcoin:master from l0rinc:l0rinc/flush-state-failure-boundary changing 2 files +33 −3
  1. l0rinc commented at 7:37 PM on July 13, 2026: contributor

    Problem: When flushing block or undo data fails, FlushStateToDisk() still writes block-index metadata and coins data, then advances m_last_flushed_block. With a separate -blocksdir, persistent failures there can leave metadata referring to block data that is not durable even when chainstate writes succeed.

    Fix: Return the flush error before those writes, leaving the last-flushed marker and on-disk coins tip unchanged in the failing call. Later flushes can retry and write if they succeed, including during shutdown. ConnectTip() and DisconnectTip() propagate the error before updating the chain tip, after applying the coins changes in memory.

    History:

    • #27866 raised whether saving other state after a flush failure was preferable. This change preserves later retries while skipping writes in the failing call.
    • #34897 introduced m_last_flushed_block to prevent persistent indexes from advancing past flushed chainstate. This fix keeps it at the last successful flush.
  2. DrahtBot added the label Validation on Jul 13, 2026
  3. DrahtBot commented at 7:37 PM on July 13, 2026: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--006a51241073e994b41acfe9ec718e94-->

    External sites

    <!--021abf342d371248e50ceaed478a90ca-->

    Reviews

    See the guideline and AI policy for information on the review process.

    Type Reviewers
    ACK optout21, arejula27, pzafonte, maflcko, ryanofsky
    Concept ACK mzumsande

    If your review is incorrectly listed, please copy-paste <code>&lt;!--meta-tag:bot-skip--&gt;</code> into the comment that the bot should ignore.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #36598 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36598.svg"></sub> ([Demo] validation: remove AssumeUTXO by l0rinc)
    • #36074 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/36074.svg"></sub> (scripted-diff: [test] Add util/check.h includes for assertions by maflcko)
    • #35731 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35731.svg"></sub> (Indexes: Harden the flush-error notification invariant by arejula27)
    • #35307 <sub><img src="https://drahtbot.space/ack_count/bitcoin/bitcoin/35307.svg"></sub> (blockstorage: keep snapshot base in normal blockfile range by shuv-amp)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

    <!--5faf32d7da4f0f540f40219e4f7537a3-->

  4. arejula27 commented at 8:13 PM on July 13, 2026: contributor

    Concept ACK

    I still have to dig into the code and the implications of this change (just did a quick review), but in the meantime: would it be worth adding a test case for the undo-file failure path as well? both the commit message and the error string mention block or undo files, but the test only injects a failure on blk*.dat.

  5. in src/test/chainstate_write_tests.cpp:124 in e1a337ee96 outdated
     119 | +    const auto* old_flushed{WITH_LOCK(::cs_main, return chainstate.GetLastFlushedBlock())};
     120 | +
     121 | +    mineBlocks(1);
     122 | +    const auto* new_tip{WITH_LOCK(::cs_main, return chainstate.m_chain.Tip())};
     123 | +    BOOST_REQUIRE_NE(old_flushed, new_tip);
     124 | +    inject_file_open_failure(WITH_LOCK(::cs_main, return chainstate.m_blockman.GetBlockPosFilename(new_tip->GetBlockPos())));
    


    l0rinc commented at 7:28 PM on July 14, 2026:

    would it be worth adding a test case for the undo-file failure path as well

    Good question, thanks - FlushBlockFile() returns false for either block or undo flush failures, see: https://github.com/bitcoin/bitcoin/blob/e1a337ee961db4308697de7cddcf4aad9684c7f1/src/node/blockstorage.cpp#L774-L786 Since this PR only changes how that shared result is handled, testing both failure sources separately would fit better in a lower-level BlockManager test.

  6. arejula27 commented at 5:19 PM on July 15, 2026: contributor

    ACK e1a337ee961db4308697de7cddcf4aad9684c7f1 This is a small change, closing a gap discussed at #34897 I built it and tested it locally. I tryied to play around it and I did not find anything wrong.

    One small, non-blocking point: the comment // FlushChainstateBlockFile() already emitted the flush-error notification. describes an invariant, but nothing checks it . It is true today only because every false path inside FlushBlockFile()/FlushUndoFile() calls flushError() first. Would be correct adding an assume to verify this behaviour is keep it in the future?:

    if (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {
        // FlushChainstateBlockFile() already emitted the flush-error notification.
        Assume(m_chainman.m_interrupt);
        return state.Error("Failed to flush block or undo file");
    }
    

    Also, I want to ask about something I found while testing, not sure if it is related to this PR. FlushStateToDisk() does not check if shutdown was already requested before it starts. I wrote a test where a first flush fails and requests shutdown, then I fix the file and call FlushStateToDisk() again: it succeeds and m_last_flushed_block moves forward, even though shutdown was already requested before this second call. Is this expected? I think it may be needed for the final flush in Shutdown() (init.cpp), but I am not 100% sure, so I wanted to ask before assuming it is fine

  7. arejula27 commented at 5:36 PM on July 15, 2026: contributor

    I opened a follow-up PR #35731 i did while reviewing the PR with the intention to group this notify + state.Error() behaviour into one helper, (making the invariant of the comment harder) and add more low-level tests like the ones you mentioned above.

  8. in src/test/chainstate_write_tests.cpp:128 in 0f04fbee2f
     123 | +    BOOST_REQUIRE_NE(old_flushed, new_tip);
     124 | +    inject_file_open_failure(WITH_LOCK(::cs_main, return chainstate.m_blockman.GetBlockPosFilename(new_tip->GetBlockPos())));
     125 | +
     126 | +    const bool flushed{chainstate.FlushStateToDisk(state, FlushStateMode::FORCE_FLUSH)};
     127 | +    BOOST_CHECK_EQUAL(m_node.exit_status.load(), EXIT_FAILURE);
     128 | +    BOOST_CHECK(flushed && state.IsValid()); // TODO: Return the flush error
    


    maflcko commented at 11:36 AM on July 16, 2026:

    nit in 0f04fbee2f123c37c4b6fbff7dc10f61dd0b5924: The && make this a bit harder to read and violate https://github.com/bitcoin/bitcoin/pull/35729


    l0rinc commented at 5:01 PM on July 16, 2026:

    flushed is tied to the state being invalid, it makes more sense to test them in groups, see #35729#pullrequestreview-4715961548

    If you insist I can remove the state validation here.


    optout21 commented at 10:28 AM on August 4, 2026:

    0f04fbe test: characterize writes after flush failure:

    Why not simply BOOST_CHECK(flushed); BOOST_CHECK(state.IsValid()); ?


    l0rinc commented at 8:59 PM on August 7, 2026:

    Yes, I could also do that, but that would assume that I find it likely that they will fail independently, and since I don't, I consider it just noise.


    maflcko commented at 8:32 AM on August 18, 2026:

    not important, but it looks like #35729 was merged, so it could make sense to follow the style used in src/test/coinscachepair_tests.cpp. Or, if you really want to group them, wrap them in another pair of (). I just mention it to avoid a silent merge conflict with #35713 and avoid having to touch this line again after merge.


    l0rinc commented at 5:19 PM on August 18, 2026:

    I gave this exact line as a concrete example for why we shouldn't just blindly split these conditions #35729#pullrequestreview-4715961548 Why not fix #35713 instead?


    optout21 commented at 4:43 AM on August 19, 2026:

    I see your argument about how unrolling is sub-optimal, but it's a weak one for me. If you insist, the proper solution may be to create a new macro with 2 (or N) conditions, BOOST_CHECK_ALL, that evaluates the conditions, and if one fails, prints the values of all. After moving away from BOOST it would be easy to add such a macro.

  9. mzumsande commented at 2:52 PM on July 16, 2026: contributor

    It would be good to discuss the real-world consequences of this change a bit more:

    As far as I understand one reason for the status quo is that even if an error would occur, and the data is not durable, there is a good chance of the fsync being done by the OS eventually if the failure is temporary - data is not necessarily lost, so we might as well flush the other state to avoid having to re-do the syncing. If the failure was permanent, the other write operations (chainstate etc.) would likely fail as well, so nothing gets out of sync in that scenario either.

    However, one point for this change is that block data is often stored separately (-blocksdir option). In this case, it is quite possible that actually corruption occurs if the blocksdir fsync fails permanently, but flushes of the chainstate etc. all succeed, and BLOCK_HAVE_DATA is set in the blocktree db, while the actual block data is missing.

    So I am Concept ACK

  10. in src/test/chainstate_write_tests.cpp:126 in 0f04fbee2f
     121 | +    mineBlocks(1);
     122 | +    const auto* new_tip{WITH_LOCK(::cs_main, return chainstate.m_chain.Tip())};
     123 | +    BOOST_REQUIRE_NE(old_flushed, new_tip);
     124 | +    inject_file_open_failure(WITH_LOCK(::cs_main, return chainstate.m_blockman.GetBlockPosFilename(new_tip->GetBlockPos())));
     125 | +
     126 | +    const bool flushed{chainstate.FlushStateToDisk(state, FlushStateMode::FORCE_FLUSH)};
    


    optout21 commented at 10:24 AM on August 4, 2026:

    0f04fbe test: characterize writes after flush failure:

    Unit test observation: Running all 781 unit tests, they all pass, but a single output line is printed on the output: Error: A fatal internal error occurred, see debug.log for details: Flushing block file to disk failed. This is likely the result of an I/O error. This is noted as this is the only stdout printout from the whole unit test suite. It is emitted from the new unit test (chainstate_flush_failure_boundary -> 2ns call to FlushStateToDisk -> FlushChainstateBlockFile -> FlushBlockFile.) Any chance to suppress this?


    l0rinc commented at 8:59 PM on August 7, 2026:

    Good point, added ASSERT_DEBUG_LOG, which suppresses the expected output and verifies the lower-level error message. It fixes it for me, can you please check it? Added you as coauthor.


    optout21 commented at 4:55 AM on August 19, 2026:

    Verified, no more output produced, thanks!

  11. in src/validation.cpp:2796 in e1a337ee96
    2791 | @@ -2792,10 +2792,9 @@ bool Chainstate::FlushStateToDisk(
    2792 |                  LOG_TIME_MILLIS_WITH_CATEGORY("write block and undo data to disk", BCLog::BENCH);
    2793 |  
    2794 |                  // First make sure all block and undo data is flushed to disk.
    2795 | -                // TODO: Handle return error, or add detailed comment why it is
    2796 | -                // safe to not return an error upon failure.
    2797 |                  if (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {
    2798 | -                    LogWarning("%s: Failed to flush block file.\n", __func__);
    2799 | +                    // FlushChainstateBlockFile() already emitted the flush-error notification.
    


    optout21 commented at 10:36 AM on August 4, 2026:

    e1a337e validation: stop writes after flush failure:

    Why is the LogWarning removed? I could not convince myself that the fatalError()/state.Error() calls include the same logging. Is it possible to preserve the log line?

    At first I though that the added comments refers to the reason of removal of LogWarning, but then I realized it explains the usage of state.Error() instead of FatalError().

    Regarding the comment, FlushChainstateBlockFile indeed calls KernelNotifications::flushError whenever it returns false, but this is handled across two levels (FlushUndoFile), which is a bit too brittle.


    l0rinc commented at 7:01 PM on August 8, 2026:

    The lower layer already reports the specific failure through flushError() before returning false, so restoring this generic warning would duplicate the error. I clarified the comment to make that explicit, let me know if it helps.

    3097:2026-08-09T00:59:53.599616Z (mocktime: 2020-08-31T15:34:13Z) [test] [noui.cpp:66] [noui_ThreadSafeMessageBoxRedirect] A fatal internal error occurred, see debug.log for details: Flushing block file to disk failed. This is likely the result of an I/O error.


    optout21 commented at 4:46 AM on August 19, 2026:

    Very good, thanks.

  12. in src/validation.cpp:2797 in e1a337ee96 outdated
    2791 | @@ -2792,10 +2792,9 @@ bool Chainstate::FlushStateToDisk(
    2792 |                  LOG_TIME_MILLIS_WITH_CATEGORY("write block and undo data to disk", BCLog::BENCH);
    2793 |  
    2794 |                  // First make sure all block and undo data is flushed to disk.
    2795 | -                // TODO: Handle return error, or add detailed comment why it is
    2796 | -                // safe to not return an error upon failure.
    2797 |                  if (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {
    2798 | -                    LogWarning("%s: Failed to flush block file.\n", __func__);
    2799 | +                    // FlushChainstateBlockFile() already emitted the flush-error notification.
    2800 | +                    return state.Error("Failed to flush block or undo file");
    


    optout21 commented at 10:55 AM on August 4, 2026:

    e1a337e validation: stop writes after flush failure:

    With this early exit, the update of m_next_write below is also skipped. It looks like that may cause some unwanted side-effects, so maybe it should be kept (e.g. by storing the return value, skipping the steps to be skipped, and keeping the if (should_write || m_next_write == ... block).


    l0rinc commented at 7:02 PM on August 8, 2026:

    I don't think m_next_write should advance after a failed write: the timer update was deliberately moved after the chainstate write in #30611. The lower layer has already requested shutdown, and the final FORCE_FLUSH during shutdown does not depend on the periodic deadline.


    optout21 commented at 4:33 AM on August 19, 2026:

    Thanks, it makes sense.


    ryanofsky commented at 5:27 PM on October 28, 2026:

    re: #35714 (review)

    Note: agree m_next_write shouldn't advance. But one minor side effect is the error can now be reported more than once.

    If the failed flush was triggered by the periodic timer or a large cache, that condition stays true. So later PERIODIC flushes (mostly from mempool acceptance) retry, fail, and call flushError() again, until connman->Interrupt() stops them. Before this PR the first attempt still wrote the state and advanced m_next_write, so the condition didn't stay true and there was only one report.

    The window is short and the node is shutting down anyway, so this seems ok.

  13. optout21 commented at 11:04 AM on August 4, 2026: contributor

    Concept ACK (e1a337ee961db4308697de7cddcf4aad9684c7f1)

    The change is within FlushStateToDisk, and changes how an error from FlushChainstateBlockFile is handled -- ignored previously, returned upwards with the change.

    Build & unit tests verfied locally.

    The effectiveness of the new test (i.e., it catches the error) is checked by manually reverting the change.

    Analyis of call pattern of FlushChainstateBlockFile: directly called in only one place, the one affected in this change. Indirectly it is used in several usecases:

    • The chain tip advances (connect/disconnect, including during IBD/reorgs),
    • A new block is accepted to disk,
    • The mempool cache is trimmed after accepting transactions/packages,
    • A user manually prunes via RPC,
    • The coins cache is resized,
    • An assumeutxo snapshot finishes background validation,
    • The node shuts down or an RPC/kernel-API caller explicitly forces a flush.

    It's hard to conceptually assess the high-level effects in all the use cases.

    Reviewing the steps excluded with the early exit, the update of the m_next_write seems uneqivocal, it may have unintended consequences (left comment).

    LGTM, left some comments.

  14. l0rinc force-pushed on Aug 8, 2026
  15. l0rinc force-pushed on Aug 8, 2026
  16. DrahtBot added the label CI failed on Aug 8, 2026
  17. DrahtBot removed the label CI failed on Aug 8, 2026
  18. l0rinc force-pushed on Aug 9, 2026
  19. l0rinc commented at 1:14 AM on August 9, 2026: contributor

    Thanks for the review, took me a bit longer to react, I'm juggling a few open ones...

  20. optout21 commented at 11:41 AM on September 14, 2026: contributor

    ACK c42d7fe1cf2aee821b4f9931ef94f16e09de8459

    Concluding my previous review. Reviewed, built and unit-tested locally. Also verified that without the behavior change, the new test fails.

  21. DrahtBot requested review from mzumsande on Sep 14, 2026
  22. l0rinc force-pushed on Oct 1, 2026
  23. l0rinc renamed this:
    validation: stop writes after flush failure
    validation: return block-file flush errors
    on Oct 4, 2026
  24. pzafonte commented at 3:06 PM on October 6, 2026: contributor

    tACK 90fd0f15b6 I'm looking into pruning for the kernel (#35627), which would go through this flush path.

  25. DrahtBot requested review from optout21 on Oct 6, 2026
  26. test: characterize writes after flush failure
    Record the flush result, last-flushed marker, block-file metadata and on-disk coins tip after a block-file flush failure.
    The failing flush still writes the new coins tip before the fix.
    
    Co-authored-by: optout <13562139+optout21@users.noreply.github.com>
    07d23eb755
  27. validation: return block-file flush errors
    Return before writing block-index metadata or coins data when the block or undo file flush fails in FlushStateToDisk().
    Keep the last-flushed marker and on-disk coins tip at the previous successful flush.
    The lower layer already reports the specific error through flushError().
    
    This stops subsequent writes in the failing invocation. Later flushes can retry and write if they succeed, including during shutdown.
    ConnectTip() and DisconnectTip() propagate the error before updating the chain tip, after applying the block's coins changes in memory.
    12f691cf28
  28. l0rinc force-pushed on Oct 7, 2026
  29. l0rinc commented at 6:13 AM on October 7, 2026: contributor

    Added missed coverage found by AI review.

  30. optout21 commented at 6:35 AM on October 7, 2026: contributor

    reACK 12f691cf286ea9a020583d267b94efca102a4056

  31. arejula27 commented at 6:58 PM on October 9, 2026: contributor

    reACK 12f691cf286ea9a020583d267b94efca102a4056

  32. pzafonte commented at 8:27 PM on October 9, 2026: contributor

    reACK 12f691cf286ea9a020583d267b94efca102a4056

  33. maflcko commented at 1:58 PM on October 12, 2026: member

    review ACK 12f691cf286ea9a020583d267b94efca102a4056 👟

    <details><summary>Show signature</summary>

    Signature:

    untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
    RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
    trusted comment: review ACK 12f691cf286ea9a020583d267b94efca102a4056 👟
    YTqlh+73BaoV9+pW0OUpkW8+H0S83QlTnH6G42LPiPC6RSvUtoVEZHtFnQr9Zx4xrcxLkrTN12hbie4ISakhCg==
    

    </details>

  34. sedited removed review request from optout21 on Oct 19, 2026
  35. sedited requested review from ryanofsky on Oct 19, 2026
  36. in src/validation.cpp:2784 in 12f691cf28
    2778 | @@ -2779,10 +2779,9 @@ bool Chainstate::FlushStateToDisk(
    2779 |                  LOG_TIME_MILLIS_WITH_CATEGORY("write block and undo data to disk", BCLog::BENCH);
    2780 |  
    2781 |                  // First make sure all block and undo data is flushed to disk.
    2782 | -                // TODO: Handle return error, or add detailed comment why it is
    2783 | -                // safe to not return an error upon failure.
    2784 |                  if (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {
    2785 | -                    LogWarning("%s: Failed to flush block file.\n", __func__);
    2786 | +                    // FlushChainstateBlockFile() already reported the specific error through flushError()
    2787 | +                    return state.Error("Failed to flush block or undo file");
    


    ryanofsky commented at 5:27 PM on October 28, 2026:

    In commit "validation: return block-file flush errors" (12f691cf286ea9a020583d267b94efca102a4056)

    Note for reviewers: this doesn't add a new kind of failure for callers. To them a false return here looks like the CheckDiskSpace() failure a few lines up, which already returns before these writes.

    Since callers were my concern in #27866 (review), here is what each one does:


    ryanofsky commented at 8:23 PM on October 28, 2026:

    In commit "validation: return block-file flush errors" (12f691cf286ea9a020583d267b94efca102a4056)

    Note: one consequence of returning here that's easy to miss. When this happens inside ConnectTip() or DisconnectTip(), the block has already been applied to CoinsTip() but m_chain isn't updated, so the coins tip and the chain tip disagree until shutdown.

    Neither ActivateBestChain() nor mempool acceptance checks the interrupt first, so a transaction or block arriving before the network threads stop could hit the asserts in PreChecks() or ConnectBlock(), and the node would abort instead of shutting down cleanly. I found this by reading the code and haven't tried to reproduce it.

    I don't think it should block this PR. It isn't corruption: the disk is still at the last successful flush, so the node recovers on restart the same way it would after a crash. And the CheckDiskSpace() return above could already get into this state.

    For a follow-up, I think a better fix than adding interrupt checks to every path that assumes the coins tip matches the chain tip is to keep the in-memory state consistent. Memory and disk each need to be consistent on their own, but disk normally lags behind memory, and a failed flush only means it lags further. It shouldn't undo a change already made in memory. So I think ConnectTip() and DisconnectTip() should ignore the flush result the way AcceptBlock() does, since the error is already reported and the block really has been applied, and let ActivateBestChain() stop on the interrupt as usual. Returning before SetTip() dates from when every block was written to disk before memory was updated; that stopped being true when flushing became periodic in #5241, but the early return stayed.

    Update: this is item 1 in the post-merge follow-ups list: #35714#pullrequestreview-5697890316

  37. in src/test/chainstate_write_tests.cpp:116 in 12f691cf28
     110 | @@ -105,4 +111,29 @@ BOOST_FIXTURE_TEST_CASE(write_during_multiblock_activation, TestChain100Setup)
     111 |      BOOST_CHECK_EQUAL(WITH_LOCK(::cs_main, return chainstate.GetLastFlushedBlock()), second_from_tip);
     112 |  }
     113 |  
     114 | +BOOST_FIXTURE_TEST_CASE(chainstate_flush_failure_boundary, TestChain100Setup)
     115 | +{
     116 | +    auto inject_file_open_failure{[](const fs::path& file) { Assert(fs::remove(file)); Assert(fs::create_directory(file)); }};
    


    ryanofsky commented at 5:27 PM on October 28, 2026:

    In commit "test: characterize writes after flush failure" (07d23eb75527b8dd6571a33b3bc304815764bc07)

    Optional, could be a follow-up: it would be nice if the test also checked what happens on the next flush after this one fails.

    With this PR, when the block file flush fails, FlushStateToDisk() returns without writing the block index or the coins. That data stays in memory, so if the problem clears up before a later FlushStateToDisk() call (say, the open failed because the node ran out of file descriptors), that call writes it. The last chance is normally the flush in Shutdown(). If the problem doesn't clear up, the node stops at the last successful flush and catches up after a restart. The first case isn't tested right now.

    Moving the block file aside instead of deleting it lets the test put it back, flush again, and check that everything catches up to new_tip:

    --- a/src/test/chainstate_write_tests.cpp
    +++ b/src/test/chainstate_write_tests.cpp
    @@ -114,5 +114,4 @@ BOOST_FIXTURE_TEST_CASE(write_during_multiblock_activation, TestChain100Setup)
     BOOST_FIXTURE_TEST_CASE(chainstate_flush_failure_boundary, TestChain100Setup)
     {
    -    auto inject_file_open_failure{[](const fs::path& file) { Assert(fs::remove(file)); Assert(fs::create_directory(file)); }};
         auto& chainstate{Assert(m_node.chainman)->ActiveChainstate()};
         BlockValidationState state{};
    @@ -124,5 +123,9 @@ BOOST_FIXTURE_TEST_CASE(chainstate_flush_failure_boundary, TestChain100Setup)
         auto* new_tip{WITH_LOCK(::cs_main, return chainstate.m_chain.Tip())};
         BOOST_REQUIRE_NE(old_flushed, new_tip);
    -    inject_file_open_failure(WITH_LOCK(::cs_main, return chainstate.m_blockman.GetBlockPosFilename(new_tip->GetBlockPos())));
    +    // Move the block file aside and put a directory in its place so the flush fails to open it.
    +    const fs::path blk_file{WITH_LOCK(::cs_main, return chainstate.m_blockman.GetBlockPosFilename(new_tip->GetBlockPos()))};
    +    const fs::path blk_backup{blk_file + ".bak"};
    +    fs::rename(blk_file, blk_backup);
    +    Assert(fs::create_directory(blk_file));
    
         ASSERT_DEBUG_LOG("Flushing block file to disk failed");
    @@ -135,4 +138,14 @@ BOOST_FIXTURE_TEST_CASE(chainstate_flush_failure_boundary, TestChain100Setup)
         BOOST_REQUIRE(WITH_LOCK(::cs_main, return chainstate.m_blockman.m_block_tree_db->ReadBlockFileInfo(new_tip->nFile, file_info)));
         BOOST_CHECK_EQUAL(file_info.nHeightLast, old_flushed->nHeight);
    +
    +    // Once the block file is back, a later flush succeeds and catches up.
    +    Assert(fs::remove(blk_file));
    +    fs::rename(blk_backup, blk_file);
    +    state = {};
    +    BOOST_CHECK(chainstate.FlushStateToDisk(state, FlushStateMode::FORCE_FLUSH));
    +    BOOST_CHECK_EQUAL(WITH_LOCK(::cs_main, return chainstate.GetLastFlushedBlock()), new_tip);
    +    BOOST_CHECK(WITH_LOCK(::cs_main, return chainstate.CoinsDB().GetBestBlock()) == new_tip->GetBlockHash());
    +    BOOST_REQUIRE(WITH_LOCK(::cs_main, return chainstate.m_blockman.m_block_tree_db->ReadBlockFileInfo(new_tip->nFile, file_info)));
    +    BOOST_CHECK_EQUAL(file_info.nHeightLast, new_tip->nHeight);
     }
    
    

    The diff is against the PR head. The retry checks pass with or without the fix (without it, the failing flush already wrote everything), so they could also go in the first commit.

    Update: this is item 5 in the post-merge follow-ups list: #35714#pullrequestreview-5697890316

  38. in src/validation.cpp:2782 in 12f691cf28
    2778 | @@ -2779,10 +2779,9 @@ bool Chainstate::FlushStateToDisk(
    2779 |                  LOG_TIME_MILLIS_WITH_CATEGORY("write block and undo data to disk", BCLog::BENCH);
    2780 |  
    2781 |                  // First make sure all block and undo data is flushed to disk.
    2782 | -                // TODO: Handle return error, or add detailed comment why it is
    2783 | -                // safe to not return an error upon failure.
    2784 |                  if (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {
    


    ryanofsky commented at 5:27 PM on October 28, 2026:

    re: #35714 (comment)

    In commit "validation: return block-file flush errors" (12f691cf286ea9a020583d267b94efca102a4056)

    I wrote a test where a first flush fails and requests shutdown, then I fix the file and call FlushStateToDisk() again: it succeeds and m_last_flushed_block moves forward, even though shutdown was already requested before this second call. Is this expected?

    Yes, this seems expected and good. The node still shuts down, but if the problem has cleared by then, the shutdown flush can save what the failed flush skipped.

    When the block file flush fails, FlushStateToDisk() now returns without writing the block index or the coins, but that data stays in memory. flushError() requests shutdown, and Shutdown() in init.cpp then calls ForceFlushStateToDisk() on each chainstate. If the problem has cleared by then (say, file descriptors released as connections close), that flush writes everything. That's what the PR description means by "Later flushes can retry and write if they succeed, including during shutdown."

    The flush doesn't check the interrupt before writing; it's checked by code that produces new state, like ActivateBestChain(). And m_last_flushed_block advancing on that flush is right: that flush succeeded, and with this PR the marker only moves on a successful flush.

  39. in src/validation.cpp:2783 in 12f691cf28
    2778 | @@ -2779,10 +2779,9 @@ bool Chainstate::FlushStateToDisk(
    2779 |                  LOG_TIME_MILLIS_WITH_CATEGORY("write block and undo data to disk", BCLog::BENCH);
    2780 |  
    2781 |                  // First make sure all block and undo data is flushed to disk.
    2782 | -                // TODO: Handle return error, or add detailed comment why it is
    2783 | -                // safe to not return an error upon failure.
    2784 |                  if (!m_blockman.FlushChainstateBlockFile(m_chain.Height())) {
    2785 | -                    LogWarning("%s: Failed to flush block file.\n", __func__);
    2786 | +                    // FlushChainstateBlockFile() already reported the specific error through flushError()
    


    ryanofsky commented at 5:27 PM on October 28, 2026:

    re: #35714 (comment)

    In commit "validation: return block-file flush errors" (12f691cf286ea9a020583d267b94efca102a4056)

    Would be correct adding an assume to verify this behaviour is keep it in the future?

    I'd leave out the Assume(m_chainman.m_interrupt), because it would fire for kernel API users on an ordinary flush failure.

    The comment promises that the notification was sent, not that shutdown was requested. In the node, flushError() does request shutdown, but in the kernel API it's just a user callback, and nothing sets the interrupt unless the application does. The node already keeps the two separate for fatalError() (m_shutdown_on_fatal_error).

    A better way to check the invariant is a test that a failed FlushBlockFile() always calls flushError(), which #35731 adds.

    Update: this comes up in items 9 and 11 of the post-merge follow-ups list: #35714#pullrequestreview-5697890316


    ryanofsky commented at 7:55 PM on October 28, 2026:

    In commit "validation: return block-file flush errors" (12f691cf286ea9a020583d267b94efca102a4056)

    Non-blocking: it would be nice if this comment also said why returning here is recoverable. Nothing in the code explains that the skipped writes are still pending in memory and can be written by a later flush. Maybe:

                        // FlushChainstateBlockFile() already reported the specific error through flushError().
                        // Return before writing the block index or coins. That data stays in memory, so a later
                        // flush can still write it if the problem clears up, for example the one during shutdown.
    

    Update: this is item 6 in the post-merge follow-ups list: #35714#pullrequestreview-5697890316

  40. in src/validation.cpp:2799 in 12f691cf28


    ryanofsky commented at 5:29 PM on October 28, 2026:

    In commit "validation: return block-file flush errors" (12f691cf286ea9a020583d267b94efca102a4056)

    Non-blocking, fine as a follow-up: if this early return happens during a prune flush and a later flush succeeds, the pruned block files are never deleted. They stay on disk until a reindex.

    How it happens:

    • FindFilesToPrune() has already marked the files pruned in memory (PruneOneBlockFile()) before the block-file flush runs, and setFilesToPrune is local to this call.
    • The later flush writes the block index with those files marked pruned, but FindFilesToPrune() skips files with nSize == 0, so nothing finds them again.

    Before this PR a block-file flush failure still unlinked them (the disk-space return above already had this gap).

    It's only leaked disk space, not corruption. A fix could keep the pending set in a member that the next successful flush drains, or there could at least be a comment here noting the gap.

    Update: this is item 7 in the post-merge follow-ups list: #35714#pullrequestreview-5697890316


    ryanofsky commented at 8:36 PM on October 28, 2026:

    In commit "validation: return block-file flush errors" (12f691cf286ea9a020583d267b94efca102a4056)

    Non-blocking: this comment says "the fatal error notification inside FlushStateToDisk still fires", but with this PR a block file flush failure is reported through flushError() instead. It would also be worth saying why ignoring the result is right. It's the same reasoning as the ConnectTip() comment: don't confuse disk state with memory state.

    -    // Flush errors (e.g. low disk space during pruning) are ignored, so that
    -    // callers can't mistreat a flush failure as a block validation failure.
    -    // The fatal error notification inside FlushStateToDisk still fires,
    -    // so the node will shut down on unrecoverable flush errors regardless.
    -    // For state a dummy value is used, and the return value is ignored.
    +    // Flush errors are ignored. Failing to save state doesn't make the block
    +    // invalid or mean accepting it should be undone, so callers shouldn't see
    +    // it as a validation failure. FlushStateToDisk() already reported the error
    +    // through flushError() or fatalError(), which normally shuts the node down.
    

    Update: this is item 6 in the post-merge follow-ups list: #35714#pullrequestreview-5697890316

  41. ryanofsky approved
  42. ryanofsky commented at 8:59 PM on October 28, 2026: contributor

    Code review ACK 12f691cf286ea9a020583d267b94efca102a4056. Thanks for putting the test first, it makes the behavior change easy to verify.

    This reverses my suggestion from #27866 (review) when that PR was trying to make the same change. But that suggestion was mainly based on not wanting to have an unjustified behavior change in a cleanup PR, and now there is a clear justification.

    Left some comments below, but none blocking. I think this should be merged given all the review it's had so far.

  43. ryanofsky assigned ryanofsky on Oct 28, 2026
  44. ryanofsky merged this on Oct 28, 2026
  45. ryanofsky closed this on Oct 28, 2026

  46. ryanofsky commented at 3:12 PM on October 29, 2026: contributor

    Post-merge, here is a list of possible follow-ups, roughly in order of importance. It collects suggestions from the review comments above, mine and other reviewers', and from follow-up PRs. Items 1-3 can crash the node or leave a corrupt database on disk; I reproduced each with a unit test. The rest are smaller bugs, documentation and tests. Happy to review any of these.

    1. ConnectTip() and DisconnectTip() should not return early on a flush failure after updating CoinsTip(). Both apply the block to CoinsTip(), then return false on a flush error before m_chain.SetTip() (disconnect, connect), so a disk error leaves the coins tip and the chain tip pointing at different blocks until shutdown.

      I checked this with a unit test: after a block file flush failure inside ConnectTip(), submitting a transaction aborts on the PreChecks() assert, and creating a block template aborts on the ConnectBlock() assert through TestBlockValidity(). So a transaction or getblocktemplate call arriving before shutdown finishes can crash the node instead of letting it shut down cleanly.

      Returning false here treats a disk failure as if it were a memory failure. Memory and disk each need to be consistent on their own, and disk normally lags behind memory; a failed flush only means it lags further. These functions should ignore the flush result the way AcceptBlock() does (the error is already reported through flushError()), and let ActivateBestChain() stop on the interrupt as usual. This predates the PR: the CheckDiskSpace() failure already returned from the same place. (original suggestion)

    2. A failed coins DB write can drop changes from memory, which can then leave a corrupt UTXO set on disk or crash the node. CCoinsViewCache::Sync(), used for periodic and prune flushes, writes the cache to the database in several batches through CCoinsViewDB::BatchWrite(). As each entry is added to a batch, the cache stops tracking it as needing a write (spent coins are erased, other entries marked clean), before the batch is committed.

      If a commit throws on a LevelDB write error, the changes in that batch are neither in the cache nor on disk, so the cache serves those spent coins from the database as unspent. FlushStateToDisk() catches the error and requests shutdown, but the node keeps running until then. If a later flush at the same block succeeds, like the shutdown flush, it writes the remaining entries and records the database as consistent at that block without the lost batch. If a block is connected first, the flush aborts on an assert instead.

      I checked this with a unit test against a real on-disk CCoinsViewDB, making one write fail partway through the flush. After the failed Sync(), 29 of 200 coins spent in the new block were visible as unspent through the cache. The successful retry then recorded the database as consistent at the new block, with those 29 spent coins still in it. A restart would load that as a valid UTXO set.

      This predates the PR and is a different code path, but it breaks the same rule as item 1: a disk failure shouldn't change memory. Keeping entries dirty until their batch commits would fix it.

    3. A failed block index write drops pending block index updates, which can leave a node that won't start without a reindex. As in item 2, WriteBlockIndexDB() erases m_dirty_fileinfo and m_dirty_blockindex before WriteBatchSync() writes them. If that write throws, the block index and file info updates are no longer marked dirty, so a later flush skips them. The shutdown flush retry can succeed, and it then writes the coins on top of a block index on disk that is missing the updates.

      I checked this with a unit test: after a failed block index write and a successful retry, the coins DB's best block was the new tip but the new tip wasn't in the block index on disk. After a simulated restart, the node failed to start with "Error initializing block database" and needed a reindex.

      Clearing the dirty sets only after the write succeeds would fix it.

    4. Document how callers should use the FlushStateToDisk() return value. The header comment only says "@returns true unless a system error occurred", which doesn't tell callers what a false return means for the state they hold. Items 1 and 8 are callers that misread it. Maybe something like:

      * [@returns](/github-metadata-backup-bitcoin-bitcoin/contributor/returns/) false if state could not be written to disk. The error has
      *     already been reported through the kernel notifications (in the node
      *     this requests shutdown).
      *
      * [@warning](/github-metadata-backup-bitcoin-bitcoin/contributor/warning/) The return value only says whether state was written to disk;
      *     in-memory state is valid either way. A caller whose own return value
      *     says whether it updated in-memory state should not return false just
      *     because this did.
      
    5. Tests that inject a failure at each write and check memory and disk afterwards. These would have caught items 1-3. Fail each write point in turn (block file, undo file, block index batch, each coins batch, including partial batches), then check that memory is still consistent (CoinsTip() best block matches m_chain.Tip(), the coins view matches the expected UTXO set) and that a restart, or a retry where that's allowed, ends up with the same state on disk as a run with no failure. A test-only option to fail the Nth database write, next to the existing simulate_crash_ratio, would make the database cases easy.

      This also covers checking that a flush after a failed one succeeds and writes the skipped state, which isn't tested yet (original suggestion). @arejula27 mentioned writing a test like this locally, so it might fit in #35731.

    6. Comments that say why the flush handling is right. The AcceptBlock() comment still refers to "the fatal error notification", and describes ignoring the result as keeping callers from mistreating a flush failure. Ignoring the result is more than defensive: callers rely on the return value to know whether the block was stored and the block index updated. If a flush error could make it return false after those updates, the return value would stop meaning anything, which would be bad for composability even though no current caller would break. Maybe something like this (original suggestion):

      // Ignore flush errors. By this point the block is stored and the block
      // index updated, and callers rely on the return value to know that.
      // Failing to write state to disk doesn't undo that, and FlushStateToDisk()
      // has already reported the error through flushError() or fatalError().
      

      Separately, the new early return in FlushStateToDisk() could say why returning is recoverable: the skipped writes stay in memory and a later flush, like the one at shutdown, can write them (original suggestion).

    7. Pruned block files aren't deleted if a prune flush fails and a later flush succeeds. FindFilesToPrune() updates the in-memory file info before the flush, but setFilesToPrune is a local, so the retry writes the index and never unlinks the files. The disk-space failure already had this gap; this PR adds the block-file failure as a second trigger. Keeping the pending set as a member that UnlinkPrunedFiles() drains after the next successful WriteBlockIndexDB() would fix it (original suggestion).

      I checked this with a unit test using -prune=1 -fastprune: after a manual prune whose flush failed and a successful retry, blk00000.dat was still there, while the same prune with no failure deletes it.

    8. ActivateBestChain() returns false after its own periodic flush fails, even though the blocks it connected are fully applied in memory. The periodic flush returns false on failure, while stopping on the interrupt returns true, so callers report a failure for blocks that were connected: the generate* RPCs say "block not accepted", and the kernel API's btck_chainstate_manager_process_block() returns an error.

      This matches the documentation ("@returns true unless a system error occurred"), but it makes false ambiguous: callers can't tell a failed disk write from a block that couldn't be connected. The disk failure is already reported through flushError(), so returning it again doesn't tell callers anything new. It should ignore the flush result like AcceptBlock() and let the interrupt check end the loop, so that false only means a block couldn't be connected or disconnected. The documentation could then say that disk write failures are reported through the notifications instead.

    9. Document that kernel API users should stop validation after a flush or fatal error. Some validation code is only safe because the node's flushError() and fatalError() handlers request shutdown, which sets the interrupt that validation checks. Kernel API users provide their own handlers, which may not request shutdown (earlier comment). The btck_NotifyFlushError and fatal error callbacks could document that the embedder should call btck_context_interrupt() and stop submitting blocks and transactions after one fires, as the node does. That is simpler than making validation correct without it, which would add code the node doesn't need.

    10. MaybeValidateSnapshot() should check its flush result before hashing the coins DB. It calls ForceFlushStateToDisk(), which only logs a failure, then hashes CoinsDB() (original suggestion). After a failed flush the DB is behind the tip, so the hash doesn't match and the snapshot chainstate is discarded as invalid. In the node this doesn't happen, because the shutdown request stops the hashing first, but that depends on the interrupt being set, not on anything this function checks.

      It should return early with a new SnapshotCompletionResult when the flush fails, without discarding anything. LoadChainstate() calls MaybeValidateSnapshot() again at the next startup, so the check is retried.

    11. #35731 (@arejula27): BlockManager-level tests and a notify-and-error helper. This covers the undo-file failure path @arejula27 asked about in this comment, which @l0rinc deferred to a lower-level test. It needs a rebase now that this is merged.

    12. RPCs that label a possibly stale coins DB with the in-memory tip. scantxoutset (code) and CreateRolledBackUTXOSnapshot() (code) ignore the ForceFlushStateToDisk() result, then read CoinsDB() and report it as m_chain.Tip(). After a failed flush the DB is at the previous flush. Minor, since the node is shutting down. If this isn't changed, the RPC help could warn that the result may lag the tip.

    13. Test style (@maflcko, @optout21). @maflcko suggested splitting the && in the test's BOOST_CHECK to follow the style from #35729 and avoid a silent conflict with the Boost removal in #35713. @optout21 suggested a BOOST_CHECK_ALL-style macro that prints every value when one condition fails. Neither change was made; this only matters when #35713 is rebased.


github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-10-30 09:40 UTC