txmempool -> policy/fees" circular dependency #13949"> txmempool -> policy/fees" circular dependency #13949"> Introduce MempoolObserver interface to break "policy/fees -> txmempool -> policy/fees" circular dependency #13949

Introduce MempoolObserver interface to break "policy/fees -> txmempool -> policy/fees" circular dependency #13949

pull Empact wants to merge 1 commits into bitcoin:master from Empact:mempool-observer changing 8 files +70 −16
  1. Empact commented at 1:09 AM on August 13, 2018: member

    A simple interface which make the implementations independent of one another.

    Note I also removed the inBlock argument to the public CBlockPolicyEstimator::removeTx, as all block-related removal calls are internal to the class. This simplifies the extracted interface and reduces CBlockPolicyEstimator surface area.

  2. fanquake added the label Refactoring on Aug 13, 2018
  3. fanquake added the label Mempool on Aug 13, 2018
  4. Empact force-pushed on Aug 13, 2018
  5. Empact force-pushed on Aug 13, 2018
  6. DrahtBot commented at 2:55 AM on August 13, 2018: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #16066 (mempool: Skip estimator if block is older than X by promag)
    • #15946 (Allow maintaining the blockfilterindex when using prune by jonasschnelli)
    • #14193 (validation: Add missing mempool locks by MarcoFalke)
    • #10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  7. DrahtBot cross-referenced this on Aug 13, 2018 from issue refactor: Removal of circular dependency between index/txindex, validation and index/base by mgrychow
  8. DrahtBot cross-referenced this on Aug 13, 2018 from issue WIP: Transaction Pool Layer by MarcoFalke
  9. practicalswift commented at 8:28 AM on August 13, 2018: contributor

    Concept ACK

    Thanks for working on these circular dependencies!

  10. in src/policy/fees.h:205 in 2a44b30ce5 outdated
     198 | @@ -198,8 +199,10 @@ class CBlockPolicyEstimator
     199 |      /** Process a transaction accepted to the mempool*/
     200 |      void processTransaction(const CTxMemPoolEntry& entry, bool validFeeEstimate);
     201 |  
     202 | -    /** Remove a transaction from the mempool tracking stats*/
     203 | -    bool removeTx(uint256 hash, bool inBlock);
     204 | +    /** Remove a transaction from the mempool tracking stats
     205 | +     * Removal is not inBlock as in-block removals are handled internally.
     206 | +     */
     207 | +    bool removeTx(uint256 hash);
    


    l2a5b1 commented at 9:37 AM on August 14, 2018:

    You can mark the declarations of interfaces::MempoolObserver methods in CBlockPolicyEstimator with override:

        void processBlock(unsigned int nBlockHeight,
                          std::vector<const CTxMemPoolEntry*>& entries) override;
    
        void processTransaction(const CTxMemPoolEntry& entry, bool validFeeEstimate) override;
    
        bool removeTx(uint256 hash) override;
    
  11. in src/policy/fees.h:202 in 2a44b30ce5 outdated
     198 | @@ -198,8 +199,10 @@ class CBlockPolicyEstimator
     199 |      /** Process a transaction accepted to the mempool*/
     200 |      void processTransaction(const CTxMemPoolEntry& entry, bool validFeeEstimate);
     201 |  
     202 | -    /** Remove a transaction from the mempool tracking stats*/
     203 | -    bool removeTx(uint256 hash, bool inBlock);
     204 | +    /** Remove a transaction from the mempool tracking stats
    


    l2a5b1 commented at 9:59 AM on August 14, 2018:

    To prevent redundant documentation, maybe you can declare the interfaces::MempoolObserver methods in CBlockPolicyEstimator without copying the documentation from the method declarations in interfaces::MempoolObserver.

        void processBlock(unsigned int nBlockHeight,
                          std::vector<const CTxMemPoolEntry*>& entries);
    
        void processTransaction(const CTxMemPoolEntry& entry, bool validFeeEstimate);
    
        /**
         * Removal is not inBlock as in-block removals are handled internally.
         */
        bool removeTx(uint256 hash);
    
  12. l2a5b1 commented at 10:27 AM on August 14, 2018: contributor

    Concept ACK. Thanks @Empact!

  13. Empact force-pushed on Aug 14, 2018
  14. Empact commented at 10:08 PM on August 14, 2018: member

    Marked methods override and made the comments more appropriate

  15. Empact force-pushed on Aug 15, 2018
  16. Empact commented at 5:42 AM on August 15, 2018: member

    Made the removeTx return type void and added interface arg comments.

  17. l2a5b1 commented at 12:07 PM on August 15, 2018: contributor

    utACK 56ec018 when squashed.

    One nit: I was wondering about the location of mempool_observer.h. If the file belongs in src/interfaces, where it currently resides, you may want to add an entry for the MempoolObserver interface to https://github.com/bitcoin/bitcoin/blob/master/src/interfaces/README.md, which contains a list of headers.

  18. Empact force-pushed on Aug 15, 2018
  19. Empact force-pushed on Aug 15, 2018
  20. Empact commented at 5:14 PM on August 15, 2018: member

    Thanks for calling out the readme. Fixed, rebased, and squashed.

  21. Empact force-pushed on Aug 15, 2018
  22. l2a5b1 commented at 6:21 PM on August 15, 2018: contributor

    yw, nice refactor! utACK f58e17e

  23. DrahtBot cross-referenced this on Aug 23, 2018 from issue Utxoscriptindex by mgrychow
  24. in src/policy/fees.cpp:512 in f58e17e386 outdated
     508 | @@ -509,6 +509,12 @@ void TxConfirmStats::removeTx(unsigned int entryHeight, unsigned int nBestSeenHe
     509 |  // tracked. Txs that were part of a block have already been removed in
     510 |  // processBlockTx to ensure they are never double tracked, but it is
     511 |  // of no harm to try to remove them again.
     512 | +void CBlockPolicyEstimator::removeTx(uint256 hash)
    


    practicalswift commented at 7:12 AM on September 5, 2018:

    hash should be passed by const reference?

  25. DrahtBot cross-referenced this on Sep 7, 2018 from issue Remove 2nd mapTx lookup in CTxMemPool::removeForBlock by promag
  26. Empact force-pushed on Sep 11, 2018
  27. Empact commented at 3:58 AM on September 11, 2018: member

    Changed removeTx and processBlock to accept const hash references.

  28. DrahtBot cross-referenced this on Sep 15, 2018 from issue Document intentional and unintentional unsigned integer overflows (wraparounds) using annotations by practicalswift
  29. DrahtBot cross-referenced this on Sep 20, 2018 from issue Log early messages with -printtoconsole by ajtowns
  30. DrahtBot cross-referenced this on Sep 21, 2018 from issue Add fee_est tool for debugging fee estimation code by ryanofsky
  31. DrahtBot cross-referenced this on Sep 27, 2018 from issue Multiprocess bitcoin by ryanofsky
  32. MarcoFalke commented at 6:26 PM on March 25, 2019: member

    copy-paste from #15638 (review) by @ryanofsky:

    While I think in some cases fixing lint-circular-dependencies errors can make code organization better, it can just as easily make it worse.

    An example would be the circular dependency between txmempool and policy/fees. The fee estimation code needs access to the mempool entry struct, and the mempool code needs to call simple fee estimator add/remove functions when mempool entries get added and removed.

    The simple and obvious way to write this code is to declare the mempool entry struct in txmempool.h and the add/remove functions in fees.h and have a circular dependency between the two object files. As long as both object files are part of the same static library, this is perfectly fine and causes no problems. But if you want to appease the lint checker, you only have bad and mediocre options:

    • Combining files that shouldn't be combined: merging fees and mempool files.
    • Splitting and scattering code around unnecessarily: declaring mempool entry struct in a separate file from the mempool struct.
    • Introducing unnecessary complexity: Registering fee estimator add/remove functions with the mempool at runtime, affecting runtime performance, and potentially introducing initialization bugs.

    It's possible changes like these would be good for other reasons, but in general I think cleaning up spew from lint-circular-dependencies is not a good reason to make a code change if you can't point to another actual reason for the change.

  33. ryanofsky commented at 7:08 PM on March 25, 2019: contributor

    re: #13949 (comment)

    Wow, I didn't know about (or maybe forgot about?) this PR. Looking over this change, it may not be a bad idea if there are justifications for it other than fixing circular dependencies. In the pasted comment, I was actually complaining circular dependency errors in the context of #10443, not this PR.

  34. MarcoFalke commented at 7:16 PM on March 25, 2019: member

    This refactoring change is +80-17. Generally I'd prefer if refactoring simplified things, not made them harder

  35. Empact force-pushed on Apr 3, 2019
  36. Empact force-pushed on Apr 3, 2019
  37. Empact commented at 4:33 AM on April 3, 2019: member

    Now +70 −16, with +43 coming from the interface header itself.

    I would argue this does make things simpler, by making CTxMemPool independent of CBlockPolicyEstimator. Both objects are now simpler because they both relate only to an abstract interface, which means there are no demands placed on either class apart from the capabilities and limitations of the interface.

    A practical benefit is that this makes CTxMemPool more flexible, for example it introduces the possibility of testing it indirectly via an observer object.

  38. l2a5b1 commented at 11:31 AM on April 3, 2019: contributor

    utACK 8501bdc. The introduction of the MempoolObserver interface seems an elegant way to break the circular dependency.

  39. DrahtBot added the label Needs rebase on Apr 10, 2019
  40. Empact force-pushed on Jun 5, 2019
  41. DrahtBot removed the label Needs rebase on Jun 5, 2019
  42. DrahtBot added the label Needs rebase on Jun 6, 2019
  43. Extract MempoolObserver interface
    This is an abstract interface which presents removes the dependency between
    CBlockPolicyEstimator and CTxMemPool
    e3e91140fc
  44. Empact force-pushed on Jun 7, 2019
  45. DrahtBot removed the label Needs rebase on Jun 7, 2019
  46. DrahtBot added the label Needs rebase on Jul 3, 2019
  47. DrahtBot commented at 2:26 PM on July 3, 2019: contributor

    <!--cf906140f33d8803c4a75a2196329ecb-->Needs rebase

  48. hebasto cross-referenced this on Dec 21, 2019 from issue refactor: Nuke policy/fees->mempool circular dependencies by hebasto
  49. Empact commented at 7:41 PM on February 20, 2020: member

    Closing in favor of #17786

  50. Empact closed this on Feb 20, 2020

  51. Empact deleted the branch on Sep 7, 2020
  52. bitcoin locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-20 06:55 UTC