[net,mempool] Call AcceptToMemoryPool() asynchronously in p2p #13413

pull skeees wants to merge 14 commits into bitcoin:master from skeees:mempool-async changing 27 files +1501 −305
  1. skeees commented at 3:22 PM on June 7, 2018: contributor

    This depends on #12934 and #13407: review those first

    As a follow up to #12934 which introduces some armature code to make message passing based communication a bit easier and moves ProcessNewBlock into its own thread, this does similar for AcceptToMemoryPool() in p2p

    If this approach seems reasonable, I think logical next steps are (in forthcoming prs) to try to reduce dependency on cs_main in p2p so that the benefits of this parallelism can be better realized

  2. skeees force-pushed on Jun 7, 2018
  3. skeees force-pushed on Jun 7, 2018
  4. DrahtBot commented at 7:33 PM on June 7, 2018: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->Note to reviewers: This pull request conflicts with the following ones:

    • #13527 (policy: Remove promiscuousmempoolflags by MarcoFalke)
    • #13481 (doc: Rewrite some validation docs as lock annotations by MarcoFalke)
    • #13426 ([bugfix] Add u8path and u8string to fix #13103 by ken2812221)
    • #13423 ([net] Thread safety annotations in net_processing by skeees)
    • #13417 ([net] Tighten scope in net_processing by skeees)
    • #13407 ([refactor, move-only-ish] Refactor mempool accept/reject logic by skeees)
    • #13399 (rpc: Add submitblockheader by MarcoFalke)
    • #13311 (Don't edit Chainparams after initialization by jtimon)
    • #13249 (Make objects in range declarations immutable by default. Avoid unnecessary copying of objects in range declarations. by practicalswift)
    • #13235 (Break circular dependency: init -> * -> init by extracting shutdown.h by Empact)
    • #13168 (Thread names in logs and deadlock debug tools (take 2) by jamesob)
    • #13145 (Use common getPath method to create temp directory in tests. by winder)
    • #13123 (net: Add Clang thread safety annotations for guarded variables in the networking code by practicalswift)
    • #13083 (Add compile time checking for cs_main runtime locking assertions by practicalswift)
    • #13076 (Fix ScanForWalletTransactions to return an enum indicating scan result: success / failure / user_abort by Empact)
    • #13029 (Interpret absense of prune= as prune=1 if there are pruned blocks by Sjors)
    • #12360 (Bury bip9 deployments by jnewbery)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  5. DrahtBot cross-referenced this on Jun 7, 2018 from issue [refactor, move-only-ish] Refactor mempool accept/reject logic by skeees
  6. DrahtBot cross-referenced this on Jun 7, 2018 from issue Document validationinterace callback blocking deadlock potential. by TheBlueMatt
  7. DrahtBot cross-referenced this on Jun 7, 2018 from issue rpc: Add submitheader by MarcoFalke
  8. DrahtBot cross-referenced this on Jun 7, 2018 from issue rpc: Avoid "duplicate" return value for invalid submitblock by MarcoFalke
  9. skeees cross-referenced this on Jun 7, 2018 from issue [net] Tighten scope in net_processing by skeees
  10. DrahtBot cross-referenced this on Jun 14, 2018 from issue rpc: Avoid "duplicate" return value for invalid submitblock by TheBlueMatt
  11. DrahtBot cross-referenced this on Jun 14, 2018 from issue [bugfix] Fix encoding issue for Windows by ken2812221
  12. DrahtBot commented at 8:08 PM on June 15, 2018: contributor

    <!--cf906140f33d8803c4a75a2196329ecb-->Needs rebase

  13. DrahtBot added the label Needs rebase on Jun 15, 2018
  14. Implement a thread-safe FIFO (producer/consumer style) queue 9668666b27
  15. Unit tests for ProducerConsumerQueue 52ac3b5469
  16. Add ConsumerThread: to consumer and operate on work from a ProducerConsumerQueue 6c624232a7
  17. ConsumerThread unit tests fb022e3850
  18. ValidationLayer() - interface for calls into block validation 5d40aa8e96
  19. Call ProcessNewBlock() asynchronously in a separate thread from p2p layer 8f0d086d53
  20. Replace all instances of ProcessNewBlock() with ValidationLayer.Validate() 0428d43f15
  21. Limit available scope of ProcessNewBlock to ValidationLayer (move-only) f31b454842
  22. Fix whitespace in test_bitcoin.cpp (whitespace,move-only) 65ae792d78
  23. [refactor, move-only-ish] Refactor mempool accept/reject logic
    Create separate functions to handle responses when a transaction
    received from a peer is accepted or rejected.
    
    This commit introduces one behavior change:
       Currently, if -promiscuousmempool is set (only allowed on regtest
       and testnet), it is possible for AcceptToMemoryPool() to return
       true, but for the CValidationState() to contain DoS points if the
       transaction failed validation with policy standard script flags
       but then passed with promiscuous flags. This will cause you to ban
       the peer who sent the transaction. This commit fixes this behavior
       by reseting the validation state in that one situation.
    d110d264c8
  24. [refactor] extract reusable logic in ValidationLayer into AsyncLayer 38f649a015
  25. ValidationLayer uses AsyncLayer 523d39f947
  26. Add MempoolLayer() for async invocation of AcceptToMemoryPool() bd24661d9f
  27. skeees force-pushed on Jun 19, 2018
  28. skeees force-pushed on Jun 19, 2018
  29. skeees force-pushed on Jun 19, 2018
  30. DrahtBot removed the label Needs rebase on Jun 19, 2018
  31. DrahtBot cross-referenced this on Jun 19, 2018 from issue doc: Rewrite some validation docs as lock annotations by MarcoFalke
  32. DrahtBot cross-referenced this on Jun 19, 2018 from issue [net] Thread safety annotations in net_processing by skeees
  33. Call AcceptToMemoryPool() asynchronously in net_processing a71183b1b0
  34. skeees force-pushed on Jun 20, 2018
  35. DrahtBot cross-referenced this on Jun 26, 2018 from issue policy: Remove promiscuousmempoolflags by MarcoFalke
  36. DrahtBot added the label Needs rebase on Jul 7, 2018
  37. DrahtBot commented at 8:37 AM on July 7, 2018: contributor

    <!--cf906140f33d8803c4a75a2196329ecb-->Needs rebase

  38. sipa referenced this in commit 1e90862f5d on Jul 14, 2018
  39. in src/core/consumerthread.h:66 in a71183b1b0
      61 | +            if (!m_threads_observed.count(id)) {
      62 | +                m_threads_observed.insert(std::this_thread::get_id());
      63 | +
      64 | +                // resubmit it so that it gets a chance to get to the right thread
      65 | +                // when resubmitting, do not block and do not care about failures
      66 | +                // theres a potential deadlock where we try to push this to a queue thats
    


    practicalswift commented at 9:17 PM on September 1, 2018:

    Typo found by codespell: theres (should be “there is”)


    practicalswift commented at 9:18 PM on September 1, 2018:

    Also: “thats” should be “that is”

  40. in src/core/producerconsumerqueue.h:128 in a71183b1b0
     123 | +    {
     124 | +        static_assert(m_consumer_mode == WorkerMode::BLOCKING, "");
     125 | +
     126 | +        T ret;
     127 | +
     128 | +        // use a temporary so theres no side effecting code inside an assert which could be disabled
    


    practicalswift commented at 9:18 PM on September 1, 2018:

    Same here: “theres” should be “there is”

  41. HashUnlimited cross-referenced this on Sep 11, 2018 from issue [net] Tighten scope in net_processing by HashUnlimited
  42. DrahtBot added the label Up for grabs on Dec 3, 2018
  43. MarcoFalke commented at 4:10 PM on December 3, 2018: member

    There hasn't been much activity lately and the patch still needs rebase, so I am closing this for now. Please let me know when you want to continue working on this, so the pull request can be re-opened.

  44. MarcoFalke closed this on Dec 3, 2018

  45. laanwj removed the label Needs rebase on Oct 24, 2019
  46. PastaPastaPasta referenced this in commit 5c6fb6e75c on Apr 15, 2020
  47. PastaPastaPasta referenced this in commit a3f0f3cf63 on Apr 16, 2020
  48. PastaPastaPasta referenced this in commit 504b696b78 on Apr 16, 2020
  49. PastaPastaPasta referenced this in commit 7e588c7594 on Apr 19, 2020
  50. PastaPastaPasta referenced this in commit 6cce7b22db on May 10, 2020
  51. PastaPastaPasta referenced this in commit abc246a698 on May 12, 2020
  52. PastaPastaPasta referenced this in commit 5b72c199ff on Jun 9, 2020
  53. PastaPastaPasta referenced this in commit 9c6d174c4d on Jun 9, 2020
  54. PastaPastaPasta referenced this in commit cfac0a9f3f on Jun 10, 2020
  55. PastaPastaPasta referenced this in commit c26f14a99c on Jun 11, 2020
  56. bitcoin locked this on Dec 16, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-19 06:54 UTC