Retire the comparison test framework #10603

issue jnewbery opened this issue on June 15, 2017
  1. jnewbery commented at 5:35 PM on June 15, 2017: member

    The comparison test framework was introduced in #5981 as a successor to the Java comparison tool, following discussion in #4545. The Java comparison tool was finally removed in #8504.

    The python comparison test framework was originally intended to start multiple bitcoind nodes on different versions, send various blocks/transactions to those nodes, and compare that the nodes reach the same state. However, the tests are now usually run through the test_runner.py framework with only one node being run. The comparison part of the comparison test framework is not even used.

    Tests written in the 'comparison' style make use of TestManager and TestInstance classes and yield statements throughout the test. The framework is overly complex and not flexible enough to be really useful. In addition, they're difficult to write, understand, update and troubleshoot. There are only 7 tests which use the comparison test style - we should rewrite them in the regular BitcoinTestFramework style.

    subtasks

    • Convert bip65-cltv-p2p.py to use the BitcoinTestFramework (#10695)
    • Convert bipdersig-p2p.py to use the BitcoinTestFramework (#10695)
    • Convert invalidtxrequest.py to use the BitcoinTestFramework (#11771)
    • Convert p2p_invalid_block.py to use the BitcoinTestFramework (#11772)
    • Convert feature_block.py to use the BitcoinTestFramework (#11773)
    • Convert feature_csv_activation.py to use the BitcoinTestFramework (#11817)
    • Delete redundant feature_bip9_softforks.py (#11818)
    • Remove the ComparisonTestFramework code (#11818)
  2. fanquake added the label Tests on Jun 16, 2017
  3. bitcoin deleted a comment on Jun 18, 2017
  4. MarcoFalke commented at 8:32 PM on June 18, 2017: member

    I'd be surprised if our tests run with a version of bitcoind older than a couple of weeks. Usually our test framework is using all "bleeding edge" features of the master branch and breaks if those features are not included in the binary. E.g. the binary of the last release.

    We would probably need to enforce compatibility, i.e. run the comparison tests as part of travis.

  5. TheBlueMatt commented at 2:46 PM on June 20, 2017: contributor

    It would be nice for the test to support running against an old bitcoind, but I see no reason why it should need to run multiple bitcoinds at once (we already support providing a path to a bitcoind, or, obviously, you can just leave an old binary in the tree).

  6. jnewbery cross-referenced this on Oct 9, 2017 from issue [tests] Make comp test framework more debuggable by jnewbery
  7. MarcoFalke referenced this in commit 76b3349159 on Nov 10, 2017
  8. jnewbery cross-referenced this on Nov 27, 2017 from issue [tests] Change invalidtxrequest to use BitcoinTestFramework by jnewbery
  9. jnewbery cross-referenced this on Nov 27, 2017 from issue [tests] Change invalidblockrequest to use BitcoinTestFramework by jnewbery
  10. jnewbery cross-referenced this on Nov 27, 2017 from issue [tests] Change feature_block.py to use BitcoinTestFramework by jnewbery
  11. jnewbery cross-referenced this on Dec 2, 2017 from issue [tests] Change feature_csv_activation.py to use BitcoinTestFramework by jnewbery
  12. jnewbery cross-referenced this on Dec 2, 2017 from issue I accidentally [deliberately] killed it [the ComparisonTestFramework] by jnewbery
  13. laanwj referenced this in commit 2dbc4a4740 on Feb 13, 2018
  14. MarcoFalke referenced this in commit 0630974647 on Mar 13, 2018
  15. laanwj referenced this in commit f0f9732d05 on Mar 29, 2018
  16. jnewbery commented at 3:41 PM on March 29, 2018: member

    #11773 merged. Next at bat: #11817.

  17. MarcoFalke referenced this in commit 2a09a78c08 on Apr 1, 2018
  18. jnewbery commented at 5:45 PM on April 2, 2018: member

    #11817 is merged. Next up: #11818

  19. MarcoFalke referenced this in commit 9a2db3b3d5 on Apr 5, 2018
  20. fanquake commented at 12:50 AM on April 5, 2018: member

    Closing now that #11818 is merged.

  21. fanquake closed this on Apr 5, 2018
  22. jnewbery commented at 2:26 PM on April 5, 2018: member

    You're too fast @fanquake - I was looking forward to closing this myself :cry:

  23. jnewbery reopened this on Apr 5, 2018
  24. jnewbery closed this on Apr 5, 2018
  25. jnewbery commented at 2:26 PM on April 5, 2018: member

    That's better

  26. MarcoFalke commented at 2:29 PM on April 5, 2018: member

    :ok_hand:

  27. HashUnlimited cross-referenced this on Aug 2, 2018 from issue [tests] Change feature_block.py to use BitcoinTestFramework by HashUnlimited
  28. HashUnlimited cross-referenced this on Aug 2, 2018 from issue [tests] Change feature_csv_activation.py to use BitcoinTestFramework by HashUnlimited
  29. HashUnlimited cross-referenced this on Aug 2, 2018 from issue I accidentally [deliberately] killed it [the ComparisonTestFramework] by HashUnlimited
  30. codablock referenced this in commit e3a7aeab7b on Sep 30, 2019
  31. codablock referenced this in commit 21040f773f on Sep 30, 2019
  32. codablock referenced this in commit 22b9bfea33 on Sep 30, 2019
  33. codablock referenced this in commit 8b88ad62b5 on Sep 30, 2019
  34. codablock referenced this in commit 33c77c4039 on Sep 30, 2019
  35. codablock referenced this in commit c32b271410 on Sep 30, 2019
  36. codablock referenced this in commit 9895e2b20e on Oct 2, 2019
  37. codablock referenced this in commit e720d73770 on Oct 2, 2019
  38. codablock referenced this in commit 79d6ae2922 on Oct 2, 2019
  39. codablock referenced this in commit dc335cdee3 on Oct 2, 2019
  40. codablock referenced this in commit d40f2e5825 on Oct 2, 2019
  41. codablock referenced this in commit 4bf88d56ce on Oct 2, 2019
  42. UdjinM6 referenced this in commit 6902ac1f2b on Oct 3, 2019
  43. UdjinM6 referenced this in commit beed5cf1f2 on Oct 3, 2019
  44. UdjinM6 referenced this in commit 400674c412 on Oct 3, 2019
  45. codablock referenced this in commit 01c138e86b on Oct 3, 2019
  46. codablock referenced this in commit 3e72a09c16 on Oct 3, 2019
  47. codablock referenced this in commit f7d1974e3d on Oct 3, 2019
  48. PastaPastaPasta referenced this in commit 128ed7f408 on Dec 27, 2019
  49. PastaPastaPasta referenced this in commit 4227b6d585 on Jan 2, 2020
  50. PastaPastaPasta referenced this in commit 12fda46dfd on Jan 4, 2020
  51. PastaPastaPasta referenced this in commit ce5b846a0f on Jan 12, 2020
  52. PastaPastaPasta referenced this in commit 107a86efe7 on Jan 12, 2020
  53. PastaPastaPasta referenced this in commit 2f4948fc71 on Jan 12, 2020
  54. PastaPastaPasta referenced this in commit 2d6af79199 on Jan 12, 2020
  55. PastaPastaPasta referenced this in commit 1fea156fa6 on Jan 12, 2020
  56. PastaPastaPasta referenced this in commit e2185c9914 on Jan 12, 2020
  57. barrystyle referenced this in commit 3598c04df5 on Jan 22, 2020
  58. barrystyle referenced this in commit 2b7f14e055 on Jan 22, 2020
  59. bitcoin locked this on Sep 8, 2021
Contributors
jnewberyMarcoFalkeTheBlueMattfanquake
Labels
Tests
Linked (view graph)
#4545 Comparison tool successor?#5981 Python P2P testing framework#8504 test: Remove java comparison tool#10695 [qa] Rewrite BIP65/BIP66 functional tests#11468 [tests] Make comp test framework more debuggable#11771 [tests] Change invalidtxrequest to use BitcoinTestFramework#11772 [tests] Change invalidblockrequest to use BitcoinTestFramework#11773 [tests] Change feature_block.py to use BitcoinTestFramework#11817 [tests] Change feature_csv_activation.py to use BitcoinTestFramework#11818 I accidentally [deliberately] killed it [the ComparisonTestFramework]
github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-20 06:55 UTC