Simplify hash.h interface using Spans #19326

pull sipa wants to merge 8 commits into bitcoin:master from sipa:202006_spanhashes changing 31 files +117 −120
  1. sipa commented at 2:29 AM on June 19, 2020: member

    This makes use of the implicit constructions and conversions to Span introduced in #18468 to simplify the hash.h interface:

    • All functions that take a pointer and a length are changed to take a Span instead.
    • The Hash() and Hash160() functions are changed to take in "range" objects instead of begin/end iterators.
  2. fanquake added the label Refactoring on Jun 19, 2020
  3. sipa force-pushed on Jun 19, 2020
  4. sipa commented at 3:53 AM on June 19, 2020: member

    @jb55 You may be interested in this.

  5. DrahtBot commented at 12:25 PM on June 19, 2020: contributor

    <!--e57a25ab6845829454e8d69fc972939a-->

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    <!--174a7506f384e20aa4161008e828411d-->

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #19628 (net: change CNetAddr::ip to have flexible size by vasild)
    • #19601 (Refactoring CHashWriter & Get{Prevouts,Sequence,Outputs}Hash to SHA256 (Alternative to #18071) by JeremyRubin)
    • #19245 ([WIP DONOTMERGE] Replace boost::filesystem with std::filesystem (in c++17) by kiminuo)
    • #19183 ([WIP DONOTMERGE] Replace boost with C++17 by MarcoFalke)
    • #18985 (bloom: use Span instead of std::vector for insert and contains [ZAP3] by jb55)
    • #18071 (Refactoring CHashWriter & Get{Prevouts,Sequence,Outputs}Hash to SHA256 by JeremyRubin)
    • #17977 (Implement BIP 340-342 validation (Schnorr/taproot/tapscript) by sipa)
    • #16841 (Replace GetScriptForWitness with GetScriptForDestination by meshcollider)
    • #10102 ([experimental] Multiprocess bitcoin by ryanofsky)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  6. DrahtBot cross-referenced this on Jun 19, 2020 from issue [WIP DONOTMERGE] Replace boost with C++17 (std::shared_mutex) by MarcoFalke
  7. DrahtBot cross-referenced this on Jun 19, 2020 from issue scripted-diff: TxoutType C++11 scoped enum class by MarcoFalke
  8. DrahtBot cross-referenced this on Jun 19, 2020 from issue bloom: use Span instead of std::vector for `insert` and `contains` [ZAP3] by jb55
  9. jb55 commented at 3:12 PM on June 19, 2020: contributor

    yup this will clean up some of my PRs. Concept ACK (I would almost say utACK but I just woke up).

  10. DrahtBot cross-referenced this on Jun 19, 2020 from issue Refactoring CHashWriter & Get{Prevouts,Sequence,Outputs}Hash to SHA256 by JeremyRubin
  11. DrahtBot cross-referenced this on Jun 19, 2020 from issue Replace current benchmarking framework with nanobench by martinus
  12. DrahtBot cross-referenced this on Jun 19, 2020 from issue Implement BIP 340-342 validation (Schnorr/taproot/tapscript) by sipa
  13. DrahtBot cross-referenced this on Jun 19, 2020 from issue Disallow automatic conversion between disparate hash types by Empact
  14. DrahtBot cross-referenced this on Jun 19, 2020 from issue Replace GetScriptForWitness with GetScriptForDestination by meshcollider
  15. DrahtBot added the label Needs rebase on Jun 21, 2020
  16. jonatack cross-referenced this on Jun 25, 2020 from issue Span improvements by sipa
  17. jonatack commented at 5:42 PM on June 25, 2020: contributor

    Concept ACK. Read the code, built/tested/running bitcoind, need to do more detailed code review stepping through the commits.

  18. practicalswift commented at 9:12 PM on June 25, 2020: contributor

    Concept ACK: Span<const unsigned char> as a drop-in replacement for const std::vector<unsigned char>& as parameter type is safe use of Span :)

  19. jonatack commented at 8:16 AM on June 26, 2020: contributor

    ACK 974381f1c53ff0674e22a40d9553f962fa0df108 a useful clear abstraction and nice cleanup. Code review, compared the implementation of as_(writable_)bytes to that shown in https://en.cppreference.com/w/cpp/container/span/as_bytes (if you retouch, s/to_/as_/ in the 45316c20 commit message). Built in various ways, ran tests and benches locally, ran bitcoind without issues for a day.

  20. sipa force-pushed on Jun 26, 2020
  21. sipa commented at 8:39 PM on June 26, 2020: member

    Rebased and addressed @jonatack's comment.

  22. DrahtBot removed the label Needs rebase on Jun 26, 2020
  23. sipa force-pushed on Jun 26, 2020
  24. sipa cross-referenced this on Jun 26, 2020 from issue span: update constructors to match c++20 draft spec and add lifetimebound attribute by theuni
  25. DrahtBot cross-referenced this on Jun 27, 2020 from issue Implement ADDRv2 support (part of BIP155) by vasild
  26. jonatack commented at 12:00 PM on June 28, 2020: contributor

    re-ACK fc7c518

  27. DrahtBot added the label Needs rebase on Jun 28, 2020
  28. sipa force-pushed on Jul 1, 2020
  29. DrahtBot removed the label Needs rebase on Jul 1, 2020
  30. DrahtBot cross-referenced this on Jul 5, 2020 from issue Replace boost::filesystem with std::filesystem by kiminuo
  31. DrahtBot cross-referenced this on Jul 7, 2020 from issue multiprocess: Add bitcoin-gui -ipcconnect option by ryanofsky
  32. DrahtBot cross-referenced this on Jul 7, 2020 from issue multiprocess: Add bitcoin-wallet -ipcconnect option by ryanofsky
  33. DrahtBot cross-referenced this on Jul 8, 2020 from issue Multiprocess bitcoin by ryanofsky
  34. jonatack commented at 3:50 AM on July 8, 2020: contributor

    Code review re-ACK 675d6fd per git range-diff abdfd2d fc7c518 675d6fd

  35. DrahtBot cross-referenced this on Jul 12, 2020 from issue Add MuHash3072 implementation by fjahr
  36. sipa cross-referenced this on Jul 16, 2020 from issue Add base_blob and BaseHash data() methods by ryanofsky
  37. in src/span.h:213 in 4679ef69f1 outdated
     207 | @@ -206,4 +208,20 @@ T& SpanPopBack(Span<T>& span)
     208 |      return back;
     209 |  }
     210 |  
     211 | +//! Mimick C++20's std::as_bytes (with unsigned char instead of std::byte)
     212 | +template<typename T>
     213 | +Span<const unsigned char> AsUnsignedChars(Span<T> in) { return {reinterpret_cast<const unsigned char*>(in.data()), in.size_bytes()}; }
    


    ryanofsky commented at 9:38 PM on July 16, 2020:

    In commit "Make uint256 Span-convertible by adding ::data()" (5d1f1d93154f3cee384e121963940141603e01f7)

    I think it'd be better to avoid doing potentially unsafe reinterpret_casts when really we only need to cast char pointers to unsigned char pointers.

    My concern is that providing this function might make it too easy to write code that looks like it is hashing vector<uint256> or Optional<uint256> or uint256* data, but actually hashing the outer pointer or container objects instead.

    I think we could provide a more limited convenience function, similar to our current CharCast function that's just restricted to chars and won't reinterpret arbitrary spans of objects.

    Experimenting a little, the following seems to work well:

    inline unsigned char* UnsignedCharCast(char* c) { return (unsigned char*)c; }
    inline unsigned char* UnsignedCharCast(unsigned char* c) { return c; }
    inline const unsigned char* UnsignedCharCast(const char* c) { return (unsigned char*)c; }
    inline const unsigned char* UnsignedCharCast(const unsigned char* c) { return c; }
    
    //! Safely convert a character array into a span of unsigned chars.
    template<typename T>
    Span<const unsigned char> UnsignedCharSpan(const T& in) { auto span = MakeSpan(in); return {UnsignedCharCast(span.data()), span.size_bytes()}; }
    

    Another advantage of this is allows replacing AsUnsignedChars(MakeSpan(in)) with simpler UnsignedCharSpan(in) everywhere.


    sipa commented at 5:01 AM on July 21, 2020:

    Agree, that seems like both a more convenient and slightly safer construction. I've implemented it slightly differently so that it remains constexpr, and supports everything MakeSpan does (including passing temporaries, and non-const return when appropriate).

    In commit "Make uint256 Span-convertible by adding ::data()" (5d1f1d9)

    I think you mean a different commit?

    My concern is that providing this function might make it too easy to write code that looks like it is hashing vector<uint256> or Optional<uint256> or uint256* data, but actually hashing the outer pointer or container objects instead.

    I don't think that would be a large concern. Those data types don't have data() and size() methods, so no implicit conversion to Span can happen. You can of course explicitly construct one, but you'd need to use something like AsUnsignedChars(Span<std::vector<uint256>(&vec, 1)), which is hopefully obviously accessing the vector's byte representation instead.

  38. in src/test/script_standard_tests.cpp:219 in b92907e0b8 outdated
     215 | @@ -216,7 +216,7 @@ BOOST_AUTO_TEST_CASE(script_standard_ExtractDestination)
     216 |      s << OP_0 << ToByteVector(pubkey.GetID());
     217 |      BOOST_CHECK(ExtractDestination(s, address));
     218 |      WitnessV0KeyHash keyhash;
     219 | -    CHash160().Write(pubkey).Finalize(keyhash.begin());
     220 | +    CHash160().Write(pubkey).Finalize(keyhash);
    


    ryanofsky commented at 10:04 PM on July 16, 2020:

    In commit "Make CHash256/CHash160 output to Span" (b92907e0b89210ecf25baa70702550f643d53525)

    This commit doesn't seem to compile. Also not sure reason scriptHash.begin() is changing to scriptHash.data() above on line 102.


    sipa commented at 5:02 AM on July 21, 2020:

    Fixed.

  39. ryanofsky approved
  40. ryanofsky commented at 10:12 PM on July 16, 2020: contributor

    Code review ACK 675d6fd044be08cd5dc7386b5574d845a74e9aae. I think intermediate commits might not compile, but comments below aren't critical.

  41. laanwj commented at 2:49 PM on July 17, 2020: member

    ACK 675d6fd044be08cd5dc7386b5574d845a74e9aae

  42. MarcoFalke added the label Waiting for author on Jul 17, 2020
  43. DrahtBot cross-referenced this on Jul 20, 2020 from issue Work around memory-aliasing in descriptor ParsePubkey by MarcoFalke
  44. sipa force-pushed on Jul 21, 2020
  45. sipa commented at 5:07 AM on July 21, 2020: member

    Rebased, and addressed @ryanofsky's comments.

  46. jonatack commented at 11:53 AM on July 21, 2020: contributor

    Code review ACK 49fc016 per git range-diff caf1766 675d6fd 49fc016. Good improvements; the main change if I'm seeing things clearly is 4679ef6 As* Spans replaced by b0cf67f9 UChar* helper functions and the refactorings they enable, and moving commit "Make script/standard's BaseHash Span-convertible" earlier in the commit order. Verified that each commit debug-builds cleanly.

  47. ryanofsky approved
  48. ryanofsky commented at 12:28 AM on July 22, 2020: contributor

    Code review ACK 49fc016c78a476e687bc73adb5c32350c2e115e5. Since last review rebased added MakeUCharSpan function, rearranged commits to fix compile error, and dropped an unneeded test change

  49. sipa removed the label Waiting for author on Jul 22, 2020
  50. DrahtBot cross-referenced this on Jul 27, 2020 from issue Refactoring CHashWriter & Get{Prevouts,Sequence,Outputs}Hash to SHA256 (Alternative to #18071) by JeremyRubin
  51. DrahtBot added the label Needs rebase on Jul 30, 2020
  52. scripted-diff: rename base_blob::data to m_data
    This is in preparation for exposing a ::data member function.
    
    -BEGIN VERIFY SCRIPT-
    sed -i "s/\([^.]\|other.\)data/\1m_data/g" src/uint256.h src/uint256.cpp
    -END VERIFY SCRIPT-
    131a2f0337
  53. Make uint256 Span-convertible by adding ::data() 567825049f
  54. Add MakeUCharSpan, to help constructing Span<[const] unsigned char>
    Based on a suggestion by Russell Yanofsky.
    e63dcc3a67
  55. Make script/standard's BaseHash Span-convertible 2a2182c387
  56. Make CHash256 and CHash160 consume Spans e549bf8a9a
  57. Make MurmurHash3 consume Spans 0ef97b1b10
  58. Make CHash256/CHash160 output to Span 02c4cc5c5d
  59. Make Hash[160] consume range-like objects 77c507358b
  60. sipa commented at 9:03 PM on July 30, 2020: member

    Rebased.

  61. sipa force-pushed on Jul 30, 2020
  62. DrahtBot removed the label Needs rebase on Jul 30, 2020
  63. DrahtBot cross-referenced this on Jul 31, 2020 from issue net: change CNetAddr::ip to have flexible size by vasild
  64. laanwj commented at 2:54 PM on August 3, 2020: member

    re-ACK 77c507358bda9bd6c496f33e0f4418c0603bb08d

  65. jonatack commented at 3:14 PM on August 3, 2020: contributor

    Code review re-ACK 77c5073 per git range-diff 14ceddd 49fc016 77c5073

  66. laanwj merged this on Aug 3, 2020
  67. laanwj closed this on Aug 3, 2020

  68. sidhujag referenced this in commit 49ec455bbf on Aug 4, 2020
  69. theStack cross-referenced this on Aug 6, 2020 from issue Per-Peer Message Capture by troygiorshev
  70. Fabcien referenced this in commit 676c36c107 on Feb 4, 2021
  71. Fabcien referenced this in commit 6d6b676c0a on Feb 4, 2021
  72. Fabcien referenced this in commit bce6951264 on Feb 4, 2021
  73. Fabcien referenced this in commit c34f1d4c2f on Feb 4, 2021
  74. Fabcien referenced this in commit f494ab3669 on Feb 4, 2021
  75. Fabcien referenced this in commit dc26954562 on Feb 4, 2021
  76. Fabcien referenced this in commit 2417468301 on Feb 4, 2021
  77. Fabcien referenced this in commit d0c313ae91 on Feb 4, 2021
  78. kwvg referenced this in commit 7f2c7ed82c on May 19, 2021
  79. kwvg referenced this in commit f7e75e3352 on May 20, 2021
  80. kwvg referenced this in commit 2dc3a1e161 on May 20, 2021
  81. kwvg referenced this in commit 94b07c2bd9 on May 20, 2021
  82. kwvg referenced this in commit 56f1b2d01c on May 20, 2021
  83. PastaPastaPasta referenced this in commit b76e7fec1f on May 21, 2021
  84. furszy cross-referenced this on Jul 6, 2021 from issue Span sources updates, up-to-date with current upstream. by furszy
  85. random-zebra referenced this in commit 56f10aaa51 on Jul 21, 2021
  86. bitcoin locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-05-19 06:53 UTC