test: Openssl removed ripemd160 #23710

MarcoFalke commented at 4:28 PM on December 8, 2021: member

Openssl removed ripemd160, see https://github.com/openssl/openssl/issues/16994

Thus, our tests fail. See below for excerpt.

Not sure how to fix this, so leaving an issue here.

Running Unit Tests for Test Framework Modules
..........
----------------------------------------------------------------------
Ran 10 tests in 0.812s

OK
4/225 - feature_block.py failed, Duration: 0 s                                                                                                                                          

stdout:


stderr:
Traceback (most recent call last):
  File "/usr/lib/python3.9/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
ValueError: [digital envelope routines] initialization error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/feature_block.py", line 54, in <module>
    from data import invalid_txs
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/data/invalid_txs.py", line 57, in <module>
    basic_p2sh = script_to_p2sh_script(CScript([OP_0]))
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/script_util.py", line 74, in script_to_p2sh_script
    return scripthash_to_p2sh_script(hash160(script))
  File "/bitcoin-core/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/script.py", line 35, in hash160
    return hashlib.new('ripemd160', sha256(s)).digest()
  File "/usr/lib/python3.9/hashlib.py", line 166, in __hash_new
    return __get_builtin_constructor(name)(data)
  File "/usr/lib/python3.9/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type ripemd160



TEST                                               | STATUS    | DURATION

wallet_backup.py --descriptors                     | ✓ Passed  | 109 s
wallet_hd.py --descriptors                         | ✓ Passed  | 92 s
wallet_hd.py --legacy-wallet                       | ✓ Passed  | 125 s
feature_block.py                                   | ✖ Failed  | 0 s

ALL                                                | ✖ Failed  | 326 s (accumulated) 
Runtime: 125 s

MarcoFalke added the label Bug on Dec 8, 2021

MarcoFalke added the label Tests on Dec 8, 2021

MarcoFalke added this to the milestone 23.0 on Dec 8, 2021

MarcoFalke commented at 4:41 PM on December 8, 2021: member

achow101 commented at 6:08 PM on December 8, 2021: member

One potential solution that I am investigating for HWI is to overwrite hashlib.new with our own implementation of it. This can then identify if the hash function is ripemd160 and return a constructor to our own implementation of ripemd160 rather than delegating to openssl. This function can fallback to the actual hashlib.new too. Since imports are global per interpreter session, modifying the imported library in one module will affect all other subsequent uses of that library.

Here is an example of how this overwrite operates:

First a new module, I calling it bitcoinhashes.py:

import hashlib

hashlib_new = hashlib.new

def our_hashlib_new(name, data=b"", **kwargs):
    print("OUR hashlib.new")
    return hashlib_new(name, data, **kwargs)

hashlib.new = our_hashlib_new

Then observe what happens when this overwritten hashlib.new is used:

Python 3.9.7 (default, Oct 25 2021, 18:20:41) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> import bitcoinhashes
>>> hashlib.new("sha256")
OUR hashlib.new
<sha256 _hashlib.HASH object @ 0x7f0367fafc70>
>>>

The ripemd160 implementation would have to be in python though. I don't think this particularly matters, but it will be way slower than openssl's, or even our own c++ implementation.

This approach is probably more suitable for the test framework rather than a library like HWI as there are no downstream consumers where changing the behavior of hashlib might be surprising.

achow101 cross-referenced this on Dec 8, 2021 from issue Replace hashlib by achow101

MarcoFalke commented at 6:29 PM on December 8, 2021: member

ripemd160 is only used in one place, so I think we can just inline ripemd160 python code for that in that place.

The issue is that no python code seems to exists, since everyone was falling back to openssl?

I don't really understand how to un-deprecate ripemd160 in openssl, but if there is an easy hack (for example via a config file), then that might be a good enough solution for the short term, also making our CI green again.

sipa commented at 6:31 PM on December 8, 2021: member

I can write a pure-python ripemd160 if that's useful.

achow101 commented at 6:33 PM on December 8, 2021: member

I can write a pure-python ripemd160 if that's useful.

It would (probably) be useful for HWI.

sipa cross-referenced this on Dec 8, 2021 from issue test: replace hashlib.ripemd160 with an own implementation by sipa

SomberNight commented at 12:03 AM on December 9, 2021: contributor

The issue is that no python code seems to exists, since everyone was falling back to openssl?

Electrum has a fallback pure-python implementation.

See https://github.com/spesmilo/electrum/blob/0df05dd914c823acae1828cad3b20bdeb13150e9/electrum/crypto.py#L312-L324 https://github.com/spesmilo/electrum/blob/0df05dd914c823acae1828cad3b20bdeb13150e9/electrum/ripemd.py

Besides the license mentioned there, I can't find where it is from exactly. It was added in https://github.com/spesmilo/electrum/commit/ad6c0ab2736edd2d7d2f0871a6b6a4d716441ace.

MarcoFalke closed this on Dec 9, 2021

sidhujag referenced this in commit 8fe77bcc20 on Dec 10, 2021

k9ert cross-referenced this on Apr 30, 2022 from issue Chore: Upgrade to python 3.10 by relativisticelectron

quartzo cross-referenced this on May 2, 2022 from issue Ubuntu 22.04 has a openssl without ripemd160 by quartzo

mmgen cross-referenced this on May 3, 2022 from issue Pycoin does not work with latest Ubuntu (22.04 Jammy) (ripemd160 has been deprecated in OpenSSL-3) by mmgen

knst referenced this in commit 2f38f567b9 on May 20, 2022

PastaPastaPasta referenced this in commit e4dbd22532 on May 29, 2022

UdjinM6 referenced this in commit 1233977a0f on May 30, 2022

PastaPastaPasta referenced this in commit 1babc5abff on May 30, 2022

windsok cross-referenced this on Jul 20, 2022 from issue ripemd160 not supported in newer versions of openssl by windsok

windsok cross-referenced this on Jul 26, 2022 from issue Add fallback to pure-python ripemd160 by windsok

xanimo cross-referenced this on Jul 27, 2022 from issue qa: backports pure python ripemd160 by xanimo

petertodd referenced this in commit de5c6a058c on Jul 29, 2022

cmdruid cross-referenced this on Sep 4, 2022 from issue Openssl removed ripemd160 support. Code using hashlib's 'ripemd160' algorithm may fail. by cmdruid

bigspider cross-referenced this on Sep 7, 2022 from issue Missing ripemd160 in hashlib by bigspider

niftynei referenced this in commit 2101ecf343 on Sep 8, 2022

niftynei cross-referenced this on Sep 8, 2022 from issue openssl: add fallback for ripemd160 implementation by niftynei

niftynei referenced this in commit e87a43d112 on Sep 8, 2022

niftynei referenced this in commit a1093f5b11 on Sep 9, 2022

shohamc1 cross-referenced this on Sep 19, 2022 from issue Replace hashlib.ripemd160 with an pure Python implementation by shohamc1

gimre-xymcity commented at 3:22 PM on September 22, 2022: none

I don't think it's worth opening new issue.

We've had similar issue, we've stripped pycryptodome from everything, leaving just ripemd: https://github.com/symbol/ripemd https://pypi.org/project/ripemd-hash/

karask cross-referenced this on Oct 31, 2022 from issue Example not working with Ubuntu 22.04 LTS by reisenmachtfreude

yashasvi-ranawat cross-referenced this on Nov 26, 2022 from issue Unsupported hash type ripemd160 by yashasvi-ranawat

waddafunk cross-referenced this on Dec 9, 2022 from issue Fix pubkey_to_address function: add native ripemd-160 by waddafunk

Fuzzbawls cross-referenced this on Mar 12, 2023 from issue [Tests] Replace hashlib.ripemd160 with native implementation by Fuzzbawls

Fuzzbawls referenced this in commit 0d779aa53d on Apr 4, 2023

mo-bay referenced this in commit 5218ea5774 on Apr 10, 2023

bitcoin locked this on Sep 22, 2023