Intermittent FAIL: test/test_bitcoin in Travis #9825

issue laanwj opened this issue on February 22, 2017

laanwj commented at 3:25 PM on February 22, 2017: member

This seems to happen at least on 0.14:

FAIL: test/test_bitcoin
=======================
test_bitcoin: /home/travis/build/bitcoin/bitcoin/depends/x86_64-unknown-linux-gnu/share/../include/boost/thread/pthread/recursive_mutex.hpp:113: void boost::recursive_mutex::lock(): Assertion `!pthread_mutex_lock(&m)' failed.
Running 228 test cases...
unknown location(0): fatal error: in "rpc_tests/rpc_rawparams": signal: SIGABRT (application abort requested)
test/rpc_tests.cpp(29): last checkpoint
test_bitcoin: key.cpp:300: void ECC_Start(): Assertion `secp256k1_context_sign == __null' failed.
unknown location(0): fatal error: in "rpc_tests/rpc_format_monetary_values": signal: SIGABRT (application abort requested)
...
*** 37 failures are detected in the test module "Bitcoin Test Suite"
test_bitcoin: /home/travis/build/bitcoin/bitcoin/depends/x86_64-unknown-linux-gnu/share/../include/boost/thread/pthread/condition_variable_fwd.hpp:102: boost::condition_variable::~condition_variable(): Assertion `!ret' failed.

Somehow it fails to lock a mutex even before the tests start, and all tests after that fail. I cannot reproduce it locally.

laanwj added the label Tests on Feb 22, 2017

laanwj commented at 7:34 AM on February 23, 2017: member

Another one: https://travis-ci.org/bitcoin/bitcoin/jobs/204158785

test_bitcoin: tpp.c:62: __pthread_tpp_change_priority: Assertion `new_prio == -1 || (new_prio >= __sched_fifo_min_prio && new_prio <= __sched_fifo_max_prio)' failed.
Running 228 test cases...
unknown location(0): fatal error: in "rpc_tests/rpc_rawparams": signal: SIGABRT (application abort requested)
test/rpc_tests.cpp(29): last checkpoint
...

laanwj cross-referenced this on Feb 23, 2017 from issue Remove block file location upgrade code by benma

laanwj commented at 7:54 AM on February 23, 2017: member

Another one on 0.14: https://travis-ci.org/bitcoin/bitcoin/jobs/204167533

FAIL: test/test_bitcoin
=======================
Running 228 test cases...
unknown location(0): fatal error: in "rpc_tests/rpc_rawsign": signal: illegal operand; address of failing instruction: 0x2b73cbf4cf3b
test/rpc_tests.cpp(29): last checkpoint

laanwj commented at 9:15 AM on February 23, 2017: member

@theuni Do you perhaps have any guess what this could be?

I think the random errors are pretty scary. I've tried running a few test_bitcoins in parallel on a trusty VM locally but never get this error. It could be that some of their testing machines have broken CPUs but that doesn't explain why it always happens in test_bitcoin and not during the RPC tests or compilation.
jnewbery cross-referenced this on Feb 23, 2017 from issue [qa] Make import-rescan.py watchonly check reliable by ryanofsky
laanwj cross-referenced this on Feb 24, 2017 from issue test_bitcoin: /usr/include/boost/thread/pthread/recursive_mutex.hpp:104: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed. by dragon28
laanwj commented at 8:51 AM on February 24, 2017: member

Apparantly this was reproduced by someone outside of travis in #9850
laanwj commented at 7:26 AM on February 27, 2017: member

There's a new wave of probably related travis failures. Here, Travis will just time out during run of make check without any output. Most likely this is test_bitcoin hanging at start.
laanwj referenced this in commit 4d60a7451b on Feb 27, 2017
laanwj cross-referenced this on Feb 27, 2017 from issue [do not merge] upload test_bitcoin executable for #9825 by laanwj
laanwj referenced this in commit 91890d3478 on Feb 27, 2017
laanwj referenced this in commit d4e00de4dd on Feb 27, 2017
ryanofsky commented at 3:27 PM on February 27, 2017: contributor

I'm not sure I understand how #9875 would fix #9825. Isn't the pthread_mutex_lock/new_prio assertion failing before any test even runs?

I do think it's likely that this bug started with #9773. As I mentioned during the IRC meeting, I was seeing these crashes first over a week ago, and looking back at travis pull request logs (https://travis-ci.org/bitcoin/bitcoin/pull_requests), the first instance of this bug that I could find was https://travis-ci.org/bitcoin/bitcoin/builds/202600549 from #9773.
laanwj commented at 2:59 PM on February 28, 2017: member

I'm not sure I understand how #9875 would fix #9825. Isn't the pthread_mutex_lock/new_prio assertion failing before any test even runs?

I haven't seen the issue since. Until I see it again, I'm not going to spend any time on it.

Also remember: messages can be out of order if one is written to stdout and the other to stderr. I think that may very well be the case here.
MarcoFalke commented at 3:09 PM on February 28, 2017: member

Closing for now, please let me know when the issue appears again.
MarcoFalke closed this on Feb 28, 2017
ryanofsky commented at 3:10 PM on February 28, 2017: contributor

I see, if the messages are out of order that makes perfect sense.
bitcoin locked this on Sep 8, 2021

Contributors

Labels

Linked (view graph)

#9773 Return errors from importmulti if complete rescans are not successful #9822 Remove block file location upgrade code #9839 [qa] Make import-rescan.py watchonly check reliable #9850 test_bitcoin: /usr/include/boost/thread/pthread/recursive_mutex.hpp:104: boost::recursive_mutex::~recursive_mutex(): Assertion `!pthread_mutex_destroy(&m)' failed.#9873 [do not merge] upload test_bitcoin executable for #9825 #9875 tests: Fix dangling pwalletMain pointer in wallet tests