WIP: Transaction Pool Layer #13804

pull MarcoFalke wants to merge 6 commits into bitcoin:master from MarcoFalke:Mf1807-txpoolStacked changing 12 files +605 −192
  1. MarcoFalke commented at 2:48 PM on July 30, 2018: member

    This implements a layer around an immutable tx pool. The layer can be seen as a temporary throw-away shell that provides the same interface as CTxMemPool. Its primary purpose right now is to be passed into ATMP while testing acceptance of several (potentially depending) transaction and then to be discarded.

    One use case could be to determine if smart contracts that are set up with multiple txs would be accepted by current consensus and policy rules.

    In the future it could be extended to support recursive wrapping of layers or a way to commit changes that happened in the layer to the underlying pool or layer. Furthermore, it could be extended to be revivable after changes to the underlying layer happened. (As opposed to be single-use)

  2. MarcoFalke added the label Validation on Jul 30, 2018
  3. DrahtBot commented at 5:42 PM on July 30, 2018: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #15681 ([mempool] Allow one extra single-ancestor transaction per package by TheBlueMatt)
    • #15192 (Add missing cs_main locks in ThreadImport(...)/Shutdown(...)/gettxoutsetinfo(...)/InitScriptExecutionCache(). Add missing locking annotations. by practicalswift)
    • #15169 (Parallelize CheckInputs() in AcceptToMemoryPool() by sdaftuar)
    • #10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  4. DrahtBot cross-referenced this on Jul 30, 2018 from issue Drop unused GetType() from CSizeComputer by Empact
  5. DrahtBot cross-referenced this on Jul 30, 2018 from issue tx pool: Use class methods to hide raw map iterator impl details by MarcoFalke
  6. MarcoFalke force-pushed on Jul 30, 2018
  7. MarcoFalke cross-referenced this on Jul 30, 2018 from issue validation: Pass tx pool reference into CheckSequenceLocks by MarcoFalke
  8. MarcoFalke cross-referenced this on Jul 30, 2018 from issue tx pool: Avoid passing redundant hash into addUnchecked (scripted-diff) by MarcoFalke
  9. in src/policy/rbf.h:26 in fa7a9de343 outdated
      22 | @@ -23,6 +23,6 @@ bool SignalsOptInRBF(const CTransaction &tx);
      23 |  // according to BIP 125
      24 |  // This involves checking sequence numbers of the transaction, as well
      25 |  // as the sequence numbers of all in-mempool ancestors.
      26 | -RBFTransactionState IsRBFOptIn(const CTransaction &tx, CTxMemPool &pool) EXCLUSIVE_LOCKS_REQUIRED(pool.cs);
      27 | +RBFTransactionState IsRBFOptIn(const CTransaction &tx,const CTxMemPool &pool) EXCLUSIVE_LOCKS_REQUIRED(pool.cs);
    


    promag commented at 2:05 PM on July 31, 2018:

    Commit "Stacked Transaction Pool"

    nit, add space after ,.

  10. in src/txmempool.cpp:353 in fa7a9de343 outdated
     351 | +void CTxMemPool::UpdateForRemoveFromMempool(const setEntries& entriesToRemove, bool updateDescendants)
     352 | +{
     353 | +    return ::UpdateForRemoveFromMempool(*this, entriesToRemove, updateDescendants);
     354 | +}
     355 |  
     356 | +void CTxMemPool::Layer::  UpdateForRemoveFromMempool(const setEntries&entriesToRemove, bool updateDescendants)
    


    promag commented at 2:06 PM on July 31, 2018:

    Commit "Stacked Transaction Pool"

    nit, remove extra spaces.

  11. promag commented at 2:08 PM on July 31, 2018: member

    Last commit is overwhelming. Could prefix with wip: until dependencies are merged?

  12. DrahtBot cross-referenced this on Aug 3, 2018 from issue validation: Document where we are intentionally ignoring bool return values from validation related functions by practicalswift
  13. MarcoFalke renamed this:
    Stacked Transaction Pool Layer
    Transaction Pool Layer
    on Aug 4, 2018
  14. MarcoFalke force-pushed on Aug 4, 2018
  15. MarcoFalke force-pushed on Aug 4, 2018
  16. MarcoFalke force-pushed on Aug 4, 2018
  17. MarcoFalke force-pushed on Aug 4, 2018
  18. MarcoFalke force-pushed on Aug 4, 2018
  19. MarcoFalke force-pushed on Aug 4, 2018
  20. MarcoFalke force-pushed on Aug 4, 2018
  21. MarcoFalke force-pushed on Aug 4, 2018
  22. MarcoFalke force-pushed on Aug 4, 2018
  23. DrahtBot cross-referenced this on Aug 8, 2018 from issue validation: Pass chainparams in AcceptToMemoryPoolWorker(...) by practicalswift
  24. DrahtBot cross-referenced this on Aug 9, 2018 from issue Lower default relay fees by ajtowns
  25. DrahtBot added the label Needs rebase on Aug 25, 2018
  26. MarcoFalke force-pushed on Aug 29, 2018
  27. MarcoFalke force-pushed on Aug 29, 2018
  28. DrahtBot removed the label Needs rebase on Aug 29, 2018
  29. MarcoFalke force-pushed on Aug 29, 2018
  30. MarcoFalke force-pushed on Aug 29, 2018
  31. MarcoFalke force-pushed on Aug 29, 2018
  32. MarcoFalke force-pushed on Aug 29, 2018
  33. MarcoFalke force-pushed on Aug 29, 2018
  34. DrahtBot cross-referenced this on Aug 31, 2018 from issue Minor style enhancement in documentation by fedsten
  35. DrahtBot added the label Needs rebase on Sep 4, 2018
  36. MarcoFalke force-pushed on Sep 7, 2018
  37. MarcoFalke force-pushed on Sep 7, 2018
  38. MarcoFalke force-pushed on Sep 7, 2018
  39. DrahtBot removed the label Needs rebase on Sep 7, 2018
  40. DrahtBot cross-referenced this on Sep 7, 2018 from issue Remove 2nd mapTx lookup in CTxMemPool::removeForBlock by promag
  41. MarcoFalke force-pushed on Sep 9, 2018
  42. MarcoFalke force-pushed on Sep 9, 2018
  43. MarcoFalke force-pushed on Sep 9, 2018
  44. MarcoFalke force-pushed on Sep 9, 2018
  45. MarcoFalke force-pushed on Sep 9, 2018
  46. laanwj referenced this in commit e0a4f9de7f on Sep 10, 2018
  47. DrahtBot added the label Needs rebase on Sep 10, 2018
  48. MarcoFalke force-pushed on Sep 10, 2018
  49. DrahtBot removed the label Needs rebase on Sep 10, 2018
  50. DrahtBot cross-referenced this on Sep 10, 2018 from issue validation: Add missing mempool locks by MarcoFalke
  51. in src/txmempool.cpp:1011 in 7bbdcd3a04 outdated
    1013 | +boost::optional<CTxMemPool::Layer::txiter> CTxMemPool::Layer::GetIter(const uint256& txid) const
    1014 |  {
    1015 | -    CTxMemPool::setEntries ret;
    1016 | +    const auto& it_cache = m_cache_added.find(txid);
    1017 | +    if (it_cache != m_cache_added.end()) {
    1018 | +        CTxMemPool::Layer::txiter{it_cache};
    


    practicalswift commented at 4:10 PM on September 14, 2018:

    Ownerless expression?

  52. DrahtBot cross-referenced this on Sep 15, 2018 from issue mempool: Fix unsigned integer wraparounds in the mempool code by practicalswift
  53. DrahtBot cross-referenced this on Sep 15, 2018 from issue Document intentional and unintentional unsigned integer overflows (wraparounds) using annotations by practicalswift
  54. practicalswift commented at 10:06 AM on September 17, 2018: contributor

    This PR fails to compile with thread safety analysis enabled when rebased on master.

  55. MarcoFalke commented at 11:01 PM on September 17, 2018: member

    @practicalswift I think clang doesn't understand "recursive" annotations. I am happy to be proven wrong, but I think this just doesn't compile with annotations for now.

  56. MarcoFalke force-pushed on Sep 19, 2018
  57. MarcoFalke force-pushed on Sep 19, 2018
  58. MarcoFalke force-pushed on Sep 19, 2018
  59. DrahtBot cross-referenced this on Sep 21, 2018 from issue Add missing locks: validation.cpp + related by practicalswift
  60. DrahtBot cross-referenced this on Sep 21, 2018 from issue Avoid unintentional unsigned integer wraparounds by practicalswift
  61. DrahtBot cross-referenced this on Sep 21, 2018 from issue Refactor: separate wallet from node by ryanofsky
  62. DrahtBot cross-referenced this on Sep 21, 2018 from issue Allow all mempool txs to be replaced after a configurable timeout (default 6h) by greenaddress
  63. DrahtBot cross-referenced this on Sep 21, 2018 from issue Add fee_est tool for debugging fee estimation code by ryanofsky
  64. in src/txmempool.h:607 in 554c3b8592 outdated
     574 | @@ -575,6 +575,9 @@ class CTxMemPool
     575 |      /** Translate a set of hashes into a set of pool iterators to avoid repeated lookups */
     576 |      setEntries GetIterSet(const std::set<uint256>& hashes) const EXCLUSIVE_LOCKS_REQUIRED(cs);
     577 |  
     578 | +    /** Dereference the txiter */
     579 | +    static const CTxMemPoolEntry& GetEntry(txiter it) { return *it; };
    


    practicalswift commented at 8:11 AM on September 23, 2018:
    2018-09-22 21:49:48 cpplint(pr=13804): src/txmempool.h:579:  You don't need a ; after a }  [readability/braces] [4]
    
  65. in src/txmempool.h:757 in 554c3b8592 outdated
     724 | +         */
     725 | +        CCriticalSection& cs;
     726 | +
     727 | +        struct txiter_nested {
     728 | +            CTxMemPool::txiter _i;
     729 | +            explicit txiter_nested(CTxMemPool::txiter val) : _i(val){};
    


    practicalswift commented at 8:11 AM on September 23, 2018:
    2018-09-22 21:49:48 cpplint(pr=13804): src/txmempool.h:729:  You don't need a ; after a }  [readability/braces] [4]
    
  66. in src/txmempool.h:758 in 554c3b8592 outdated
     725 | +        CCriticalSection& cs;
     726 | +
     727 | +        struct txiter_nested {
     728 | +            CTxMemPool::txiter _i;
     729 | +            explicit txiter_nested(CTxMemPool::txiter val) : _i(val){};
     730 | +            txiter_nested() : _i(){};
    


    practicalswift commented at 8:12 AM on September 23, 2018:
    2018-09-22 21:49:48 cpplint(pr=13804): src/txmempool.h:730:  You don't need a ; after a }  [readability/braces] [4]
    
  67. in src/txmempool.h:759 in 554c3b8592 outdated
     754 | +        using txlinksMap = std::map<CTxMemPool::Layer::txiter, CTxMemPool::Layer::TxLinks, CompareIteratorByHash>;
     755 | +        /**
     756 | +         * Construct a Layer based on an existing tx pool.
     757 | +         * The caller must acquire the lock for the whole life-time of this layer.
     758 | +         */
     759 | +        Layer(const CTxMemPool& p) EXCLUSIVE_LOCKS_REQUIRED(m_tx_pool.cs)
    


    practicalswift commented at 8:12 AM on September 23, 2018:
    2018-09-22 21:49:48 cpplint(pr=13804): src/txmempool.h:759:  Single-parameter constructors should be marked explicit.  [runtime/explicit] [5]
    
  68. in src/txmempool.h:808 in 554c3b8592 outdated
     803 | +        void UpdateChildrenForRemoval(txiter it);
     804 | +        const setEntries& GetMemPoolChildren(txiter entry) const;
     805 | +        void removeUnchecked(txiter it, MemPoolRemovalReason reason);
     806 | +
     807 | +    private:
     808 | +
    


    practicalswift commented at 8:13 AM on September 23, 2018:
    2018-09-22 21:49:48 cpplint(pr=13804): src/txmempool.h:808:  Do not leave a blank line after "private:"  [whitespace/blank_line] [3]
    
  69. in src/txmempool.cpp:468 in 554c3b8592 outdated
     469 | -    vTxHashes.emplace_back(tx.GetWitnessHash(), newit);
     470 | +    vTxHashes.emplace_back(entry.GetTx().GetWitnessHash(), newit);
     471 |      newit->vTxHashesIdx = vTxHashes.size() - 1;
     472 |  }
     473 |  
     474 | +void CTxMemPool::Layer::addUnchecked(const CTxMemPoolEntry& entry, const CTxMemPool::Layer::setEntries& setAncestors, bool /* validFeeEstimate */ ignore)
    


    practicalswift commented at 5:13 AM on September 26, 2018:

    Use an unnamed parameter instead of ignore.

  70. in src/txmempool.cpp:474 in 554c3b8592 outdated
     475 | +{
     476 | +    ::addUnchecked(*this, entry, setAncestors);
     477 | +}
     478 | +
     479 | +template <typename P>
     480 | +static void removeUnchecked(P& pool, typename P::txiter it, MemPoolRemovalReason reason)
    


    practicalswift commented at 5:13 AM on September 26, 2018:

    Make MemPoolRemovalReason an unnamed parameter.

  71. in src/txmempool.cpp:898 in 554c3b8592 outdated
     903 | +    auto it = m_cache_added.find(hash);
     904 | +    if (it == m_cache_added.end())
     905 | +        return nullptr;
     906 | +    else
     907 | +        return it->GetSharedTx();
     908 | +    if (m_cache_removed.find(hash) != m_cache_removed.end()) return nullptr;
    


    practicalswift commented at 5:14 AM on September 26, 2018:

    This code will never get executed.

  72. in src/txmempool.cpp:1139 in 554c3b8592 outdated
    1135 | @@ -937,14 +1136,31 @@ int CTxMemPool::Expire(int64_t time) {
    1136 |      RemoveStaged(stage, false, MemPoolRemovalReason::EXPIRY);
    1137 |      return stage.size();
    1138 |  }
    1139 | +int CTxMemPool::Layer::Expire(int64_t time)
    


    practicalswift commented at 5:15 AM on September 26, 2018:

    Make time an unnamed parameter.

  73. in src/txmempool.cpp:1319 in 554c3b8592 outdated
    1315 | @@ -1058,6 +1316,13 @@ void CTxMemPool::TrimToSize(size_t sizelimit, std::vector<COutPoint>* pvNoSpends
    1316 |      }
    1317 |  }
    1318 |  
    1319 | +void CTxMemPool::Layer::TrimToSize(size_t sizelimit, std::vector<COutPoint>* pvNoSpendsRemaining)
    


    practicalswift commented at 5:16 AM on September 26, 2018:

    Make sizelimit and pvNoSpendsRemaining unnamed parameters.

  74. MarcoFalke force-pushed on Sep 26, 2018
  75. DrahtBot cross-referenced this on Sep 27, 2018 from issue Multiprocess bitcoin by ryanofsky
  76. DrahtBot cross-referenced this on Oct 9, 2018 from issue Remove redundant run time assertions for locks already checked at compile time by practicalswift
  77. MarcoFalke force-pushed on Oct 27, 2018
  78. MarcoFalke renamed this:
    Transaction Pool Layer
    WIP: Transaction Pool Layer
    on Oct 28, 2018
  79. MarcoFalke force-pushed on Oct 28, 2018
  80. AM5800 cross-referenced this on Oct 29, 2018 from issue Dandelion (BIP 156) by thothd
  81. in src/txmempool.cpp:1000 in f1d3f1b93a outdated
     994 | +{
     995 | +    auto it_add = m_map_next_tx_added.find(prevout);
     996 | +    if (it_add != m_map_next_tx_added.end()) return it_add->second;
     997 | +    if (m_map_next_tx_removed.find(prevout) != m_map_next_tx_removed.end()) return nullptr;
     998 | +
     999 | +    return m_tx_pool.GetConflictTx(prevout);
    


    practicalswift commented at 12:25 PM on December 8, 2018:

    Calling GetConflictTx requires holding the mutex m_tx_pool.cs

  82. in src/txmempool.cpp:1021 in f1d3f1b93a outdated
    1017 | +    if (m_cache_removed.find(txid) != m_cache_removed.end()) {
    1018 | +        return boost::optional<CTxMemPool::Layer::txiter>{};
    1019 | +    }
    1020 | +
    1021 | +    // Nothing found in the cache, fall back to the inner layer:
    1022 | +    const auto it_inner = m_tx_pool.GetIter(txid);
    


    practicalswift commented at 12:26 PM on December 8, 2018:

    m_tx_pool.cs must be held when calling GetIter

  83. in src/txmempool.cpp:1222 in f1d3f1b93a outdated
    1210 | +{
    1211 | +    const auto it = m_cache_map_links.find(entry);
    1212 | +    if (it != m_cache_map_links.end()) return it->second.parents;
    1213 | +
    1214 | +    CTxMemPool::Layer::setEntries ret;
    1215 | +    for (const auto& parent : m_tx_pool.GetMemPoolParents(boost::get<CTxMemPool::txiter>(entry))) {
    


    practicalswift commented at 12:26 PM on December 8, 2018:

    m_tx_pool.cs must be held here

  84. practicalswift commented at 12:28 PM on December 8, 2018: contributor

    This PR doesn't compile due to missing locks

  85. DrahtBot added the label Needs rebase on Dec 13, 2018
  86. MarcoFalke force-pushed on Dec 22, 2018
  87. MarcoFalke force-pushed on Dec 22, 2018
  88. DrahtBot removed the label Needs rebase on Dec 24, 2018
  89. DrahtBot added the label Needs rebase on Jan 15, 2019
  90. MarcoFalke force-pushed on Mar 7, 2019
  91. DrahtBot removed the label Needs rebase on Mar 7, 2019
  92. MarcoFalke force-pushed on Mar 8, 2019
  93. DrahtBot added the label Needs rebase on Mar 21, 2019
  94. MarcoFalke force-pushed on Jul 10, 2019
  95. DrahtBot removed the label Needs rebase on Jul 10, 2019
  96. MarcoFalke force-pushed on Jul 11, 2019
  97. MarcoFalke force-pushed on Jul 11, 2019
  98. MarcoFalke force-pushed on Jul 11, 2019
  99. MarcoFalke force-pushed on Jul 11, 2019
  100. MarcoFalke force-pushed on Jul 11, 2019
  101. MarcoFalke force-pushed on Jul 11, 2019
  102. txpool: Avoid mapTx.iterator_to lookup in CalculateMemPoolAncestors b33a0fb7f9
  103. tx pool: Make pool a template parameter of CoinsViewMemPool a6bf4f0588
  104. tx pool: Move ATMP internal logic into static functions for template-reusability 980353a186
  105. txmempool: Use auto and GetEntry to hide txiter type 19c277055c
  106. MarcoFalke force-pushed on Jul 13, 2019
  107. Transaction Pool Layer f57817fca0
  108. rpc: Use TxPoolLayer in testmempoolaccept 421de16e9c
  109. MarcoFalke force-pushed on Jul 13, 2019
  110. MarcoFalke closed this on Jul 16, 2019

  111. MarcoFalke deleted the branch on Jul 16, 2019
  112. MarcoFalke cross-referenced this on Jul 16, 2019 from issue rpc: testmempoolaccept for list of transactions by MarcoFalke
  113. Munkybooty referenced this in commit 208626a527 on Jul 7, 2021
  114. 5tefan referenced this in commit 63fa38955b on Aug 10, 2021
  115. 5tefan referenced this in commit 7b6837b117 on Aug 11, 2021
  116. 5tefan referenced this in commit f719cafdcd on Aug 12, 2021
  117. 5tefan referenced this in commit 495cfa67f2 on Aug 12, 2021
  118. 5tefan referenced this in commit db302f912d on Aug 12, 2021
  119. bitcoin locked this on Dec 16, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-19 06:54 UTC