wallet: Never schedule MaybeCompactWalletDB when -flushwallet is off #18923

pull MarcoFalke wants to merge 5 commits into bitcoin:master from MarcoFalke:2005-walletNoFlush changing 13 files +45 −27
  1. MarcoFalke commented at 1:12 PM on May 9, 2020: member

    User-facing, this is a refactor. Internally, the scheduler does not have to call a mostly empty function every half a second.

  2. MarcoFalke commented at 1:13 PM on May 9, 2020: member

    Can be tested by applying the following diff and observing a failing test:

    diff --git a/test/functional/wallet_dump.py b/test/functional/wallet_dump.py
    index 6bfb468823..f30589761a 100755
    --- a/test/functional/wallet_dump.py
    +++ b/test/functional/wallet_dump.py
    @@ -190,7 +190,7 @@ class WalletDumpTest(BitcoinTestFramework):
             assert_raises_rpc_error(-8, "already exists", lambda: self.nodes[0].dumpwallet(wallet_enc_dump))
     
             # Restart node with new wallet, and test importwallet
    -        self.restart_node(0, ['-wallet=w2'])
    +        self.restart_node(0, ['-wallet=w2', '-flushwallet=0'])
     
             # Make sure the address is not IsMine before import
             result = self.nodes[0].getaddressinfo(multisig_addr)
    
  3. MarcoFalke added the label Refactoring on May 9, 2020
  4. MarcoFalke added the label Wallet on May 9, 2020
  5. MarcoFalke force-pushed on May 9, 2020
  6. MarcoFalke force-pushed on May 9, 2020
  7. MarcoFalke force-pushed on May 9, 2020
  8. MarcoFalke force-pushed on May 9, 2020
  9. DrahtBot commented at 3:16 AM on May 10, 2020: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #19461 (multiprocess: Add bitcoin-gui -ipcconnect option by ryanofsky)
    • #19460 (multiprocess: Add bitcoin-wallet -ipcconnect option by ryanofsky)
    • #19335 (wallet: Cleanup and separate BerkeleyDatabase and BerkeleyBatch by achow101)
    • #19101 (refactor: remove ::vpwallets and related global variables by ryanofsky)
    • #19099 (refactor: Move wallet methods out of chain.h and node.h by ryanofsky)
    • #19098 (test: Remove duplicate NodeContext hacks by ryanofsky)
    • #18904 (Don't call lsn_reset in periodic flush by bvbfan)
    • #18354 (Use shared pointers only in validation interface by bvbfan)
    • #15937 (Add loadwallet and createwallet load_on_startup options by ryanofsky)
    • #10102 ([experimental] Multiprocess bitcoin by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  10. DrahtBot cross-referenced this on May 10, 2020 from issue wallet: Keep inactive seeds after sethdseed and derive keys from them as needed by achow101
  11. MarcoFalke force-pushed on May 11, 2020
  12. DrahtBot added the label Needs rebase on May 22, 2020
  13. MarcoFalke force-pushed on May 22, 2020
  14. MarcoFalke force-pushed on May 22, 2020
  15. DrahtBot removed the label Needs rebase on May 22, 2020
  16. DrahtBot cross-referenced this on May 28, 2020 from issue Remove g_rpc_chain global by ryanofsky
  17. MarcoFalke cross-referenced this on May 28, 2020 from issue test: Remove duplicate NodeContext hacks by ryanofsky
  18. DrahtBot cross-referenced this on May 29, 2020 from issue refactor: remove ::vpwallets and related global variables by ryanofsky
  19. DrahtBot cross-referenced this on May 29, 2020 from issue refactor: Move wallet methods out of chain.h and node.h by ryanofsky
  20. in src/interfaces/wallet.cpp:494 in faccdaba58 outdated
     490 | @@ -491,7 +491,7 @@ class WalletClientImpl : public ChainClient
     491 |      }
     492 |      bool verify() override { return VerifyWallets(m_chain, m_wallet_filenames); }
     493 |      bool load() override { return LoadWallets(m_chain, m_wallet_filenames); }
     494 | -    void start(CScheduler& scheduler) override { return StartWallets(scheduler); }
     495 | +    void start(CScheduler& scheduler, const ArgsManager& argsman) override { return StartWallets(scheduler, argsman); }
    


    ryanofsky commented at 11:49 AM on May 29, 2020:

    In commit "wallet: Pass unused argsman to StartWallets()" (faccdaba583288273399481b1f7892a910080fda)

    Not a big deal and I can fix it later, but this is a bit broken with #10102 because it is trying to pass a whole argsman object from the node to wallet process just to pass a single bool. (See "Interface method parameter and return types should either be serializable or be other interface classes" examples in https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#internal-interface-guidelines)

    Recommended fix for this would be to pass -flushwallet bool setting to StartWallets the same way the -wallet list setting gets passed to LoadWallets above: first read in WalletInit::Construct and passed as an argument to MakeWalletClient, stored in the WalletClientImpl instance, and used as needed


    MarcoFalke commented at 12:49 PM on May 29, 2020:

    Thanks, tried to implement your suggestion in branch 9ddddd2d927b3a752e4698c4771fc68cdc2d9698


    ryanofsky commented at 1:33 PM on May 29, 2020:

    Thanks, tried to implement your suggestion in branch 9ddddd2

    Branch looks good. Current version is fine too in case you prefer that or the suggestion doesn't work.

    I was thinking maybe the best another approach after #19096 would be to add an ArgsManager pointer to the WalletContext struct, and initializing it from a new MakeWalletClient ArgsManager& argument so the wallet filenames and periodic flush arguments and temporary storage could be dropped


    MarcoFalke commented at 5:21 PM on June 5, 2020:

    add an ArgsManager pointer to the WalletContext struct

    thx, done

  21. ryanofsky approved
  22. ryanofsky commented at 11:57 AM on May 29, 2020: contributor

    Code review ACK fa6195467a3533fcdcc001d559555a86717cd5d3

  23. MarcoFalke force-pushed on May 29, 2020
  24. ryanofsky approved
  25. ryanofsky commented at 1:40 PM on May 29, 2020: contributor

    Code review ACK 9ddddd2d927b3a752e4698c4771fc68cdc2d9698. SInce last review was changed to pass periodic flush option from WalletInit::Construct

  26. DrahtBot cross-referenced this on May 29, 2020 from issue Use shared pointers only in validation interface by bvbfan
  27. DrahtBot cross-referenced this on Jun 2, 2020 from issue Multiprocess bitcoin by ryanofsky
  28. DrahtBot added the label Needs rebase on Jun 5, 2020
  29. MarcoFalke force-pushed on Jun 5, 2020
  30. DrahtBot removed the label Needs rebase on Jun 5, 2020
  31. MarcoFalke force-pushed on Jun 5, 2020
  32. in src/wallet/test/init_test_fixture.h:21 in fa00c37cba outdated
      17 | @@ -18,7 +18,6 @@ struct InitWalletDirTestingSetup: public BasicTestingSetup {
      18 |      fs::path m_datadir;
      19 |      fs::path m_cwd;
      20 |      std::map<std::string, fs::path> m_walletdir_path_cases;
      21 | -    NodeContext m_node;
    


    ryanofsky commented at 6:27 PM on June 5, 2020:

    In commit "test: Avoid overwriting the NodeContext member of the testing setup" (fa00c37cba98794c1c12c4b77e885d831d9bf183)

    Wow, good catches. I wonder if there was a compiler warning we could enable that would have caught these.

  33. in src/util/check.h:47 in fa38755d2f outdated
      41 | @@ -42,4 +42,12 @@ class NonFatalCheckError : public std::runtime_error
      42 |          }                                                         \
      43 |      } while (false)
      44 |  
      45 | +/** Assert that the passed pointer is not nullptr and return a reference to the pointed object */
      46 | +template <typename PtrT>
      47 | +typename std::remove_reference<decltype(*PtrT{})>::type& Ensure(const PtrT& ptr)
    


    ryanofsky commented at 6:46 PM on June 5, 2020:

    In commit "wallet: Pass unused args to WalletClientImpl" (fa38755d2fc444589ae506e4edd8597ab25e55e9)

    Slight preference for calling this AssertNotNull instead of Ensure just because it could wind up being used widely, and it'd be nice for it to be obvious what it does.

    Also for code review purposes, C++ programmers are already pretty trained to know that *ptr is dangerous and can kill the program, while Ensure(ptr) might look more innocuous if it's part of a larger expression. And since the name doesn't contain "Assert" or "Abort", it might actually be clearer if this returned a pointer so the caller is forced to write *. Could also go more generic and provide

    template<typename T> T&& Assert(T&& value) { assert(value); return std::forward<T>(value); }
    
  34. ryanofsky approved
  35. ryanofsky commented at 6:51 PM on June 5, 2020: contributor

    Code review ACK 00000d48588689f7576a8f4cbd9eb8edee3b53b0. No important suggestions. First two commits are the same since last review, last commit is basically the same, middle commits switch back to passing ArgsManager instead of the bool, but going through WalletContext struct from #19096

  36. fanquake commented at 6:06 AM on June 6, 2020: member

    Concept ACK

  37. MarcoFalke force-pushed on Jun 8, 2020
  38. MarcoFalke commented at 12:42 PM on June 8, 2020: member

    commit id 00000d4 was lost in a rebase

  39. DrahtBot cross-referenced this on Jun 9, 2020 from issue Overhaul transaction request logic by sipa
  40. ryanofsky approved
  41. ryanofsky commented at 4:45 PM on June 9, 2020: contributor

    Code review ACK faf18d6c86cbb3e90070a971ee341cc76fb1a8f5. Changes since previous review were replacing Ensure with Assert, and adding new checks to ensure the NDEBUG is disabled and that assert isn't called from RPC code. First two commits are new, previous shadow fix commit became #19188, and the other commits are basically unchanged except being divided differently and replacing assert with ensure. Long list. Nice cleanups, though

  42. in src/util/check.h:51 in fa54cf1073 outdated
      45 | @@ -46,4 +46,12 @@ class NonFatalCheckError : public std::runtime_error
      46 |  #error "Cannot compile without assertions!"
      47 |  #endif
      48 |  
      49 | +/** Identity function. Abort if the value compares equal to zero */
      50 | +template <typename T>
      51 | +T&& Assert(T&& value)
    


    ryanofsky commented at 5:15 PM on June 9, 2020:

    In commit "util: Add Assert identity function" (fa54cf107319609ff44de960b987bd09b46945d0)

    Thinking about this, I think it would be better to change signature to T Assert(T&& value) to avoid a possibility of returning a reference to a temporary. Contrived example but I think p would be an invalid reference here returning T&& instead of T:

    std::unique_ptr<int>&& p = Assert(MakeUnique<int>(5));
    

    No other change should be needed to fix this other than removing && from return type.

  43. MarcoFalke force-pushed on Jun 9, 2020
  44. MarcoFalke commented at 6:50 PM on June 14, 2020: member

    I've split out the first commits into #19277

  45. DrahtBot cross-referenced this on Jun 15, 2020 from issue util: Add Assert identity function by MarcoFalke
  46. DrahtBot cross-referenced this on Jun 20, 2020 from issue build: Do not include server symbols in wallet by MarcoFalke
  47. ryanofsky approved
  48. ryanofsky commented at 7:53 PM on June 23, 2020: contributor

    Would be good to merge #19277 first, but code review ACK fac173681bfcd0debd7f203aeb4eb6910407cce8. Only change since last review is suggested change to assert return type

  49. DrahtBot cross-referenced this on Jun 25, 2020 from issue Remove the automatic creation and loading of the default wallet by achow101
  50. DrahtBot cross-referenced this on Jun 25, 2020 from issue refactor: Drop g_orphan_list global by hebasto
  51. DrahtBot cross-referenced this on Jun 26, 2020 from issue Add loadwallet and createwallet load_on_startup options by ryanofsky
  52. DrahtBot added the label Needs rebase on Jul 1, 2020
  53. MarcoFalke force-pushed on Jul 4, 2020
  54. DrahtBot removed the label Needs rebase on Jul 4, 2020
  55. MarcoFalke commented at 6:01 PM on July 4, 2020: member

    Rebased

  56. DrahtBot cross-referenced this on Jul 7, 2020 from issue multiprocess: Add bitcoin-gui -ipcconnect option by ryanofsky
  57. DrahtBot cross-referenced this on Jul 7, 2020 from issue multiprocess: Add bitcoin-wallet -ipcconnect option by ryanofsky
  58. test: Add smoke test to check that wallets are flushed by default fa28a61897
  59. gui tests: Limit life-time of dummy testing setup
    Its only purpose is to create a directory. So instead of keeping it
    alive, use it only to get the path of the directory and then create it
    explicitly.
    fa6c186436
  60. wallet: Pass unused args to StartWallets
    This refactor does not change behavior
    faf8401c19
  61. wallet: Never schedule MaybeCompactWalletDB when -flushwallet is off fa7b164d62
  62. refactor: Use C++11 range-based for loop fa73493930
  63. MarcoFalke cross-referenced this on Jul 9, 2020 from issue Refactor: clean up PeriodicFlush() by jnewbery
  64. MarcoFalke force-pushed on Jul 9, 2020
  65. in src/wallet/bdb.cpp:627 in fa73493930
     622 | @@ -623,8 +623,8 @@ bool BerkeleyDatabase::PeriodicFlush()
     623 |      if (!lockDb) return false;
     624 |  
     625 |      // Don't flush if any databases are in use
     626 | -    for (auto it = env->mapFileUseCount.begin() ; it != env->mapFileUseCount.end(); it++) {
     627 | -        if ((*it).second > 0) return false;
     628 | +    for (const auto& use_count : env->mapFileUseCount) {
     629 | +        if (use_count.second > 0) return false;
    


    promag commented at 11:14 AM on July 9, 2020:

    fa73493930e35850e877725167dc9d42e47015c8

    nit, if you feel like cleaning it a bit more maybe do this below:

    if (env->mapFileUseCount.erase(strFile) == 0) return false
    

    And drop .erase(it).


    MarcoFalke commented at 11:21 AM on July 9, 2020:

    Thanks, but I'll leave that for a follow-up

  66. in test/functional/wallet_dump.py:206 in fa73493930
     201 | @@ -202,5 +202,10 @@ def run_test(self):
     202 |          result = self.nodes[0].getaddressinfo(multisig_addr)
     203 |          assert result['ismine']
     204 |  
     205 | +        self.log.info('Check that wallet is flushed')
     206 | +        with self.nodes[0].assert_debug_log(['Flushing wallet.dat'], timeout=20):
    


    jnewbery commented at 11:45 AM on July 9, 2020:

    I know you'll disagree, but I think it's always better to test the result of an action rather than whether it was logged. In this case, I think you could test the file modification time before and after.


    MarcoFalke commented at 12:16 PM on July 9, 2020:

    I think both can be done at the same time, but asserting the modification time increases requires a time.sleep(1), since os.path.getmtime returns seconds. I don't like sleeps longer than 200 ms in tests, so I'll leave that for a follow-up ;)


    jnewbery commented at 12:45 PM on July 9, 2020:

    No need for a follow-up. Fast tests are nice too!

  67. jnewbery commented at 12:02 PM on July 9, 2020: member

    utACK fa73493930e35850e877725167dc9d42e47015c8

    One comment on the new test. Feel free to ignore.

  68. DrahtBot cross-referenced this on Jul 9, 2020 from issue Don't call lsn_reset in periodic flush by bvbfan
  69. DrahtBot cross-referenced this on Jul 10, 2020 from issue wallet: Cleanup and separate BerkeleyDatabase and BerkeleyBatch by achow101
  70. meshcollider commented at 10:51 AM on July 11, 2020: contributor

    utACK fa73493930e35850e877725167dc9d42e47015c8

  71. ryanofsky approved
  72. ryanofsky commented at 11:08 AM on July 11, 2020: contributor

    Code review ACK fa73493930e35850e877725167dc9d42e47015c8. Just rebased since last review

  73. meshcollider merged this on Jul 11, 2020
  74. meshcollider closed this on Jul 11, 2020

  75. MarcoFalke deleted the branch on Jul 11, 2020
  76. sidhujag referenced this in commit c96248b149 on Jul 11, 2020
  77. hebasto cross-referenced this on Jul 24, 2020 from issue util: Make default arg values more specific by hebasto
  78. Fabcien referenced this in commit 912bd27d4b on Dec 9, 2020
  79. Brightside56 cross-referenced this on Apr 22, 2021 from issue Add -flushwalletinterval command line arg by Brightside56
  80. bitcoin locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-20 06:54 UTC