RPC tests are failing to start bitcoind #9086

On master I'm getting this:

every time I run rpc tests batched with multiple threads(and sometimes with 1), certain tests basically fail to start: Unexpected exception caught during testing: Exception('bitcoind exited with status 1 during initialization',)

Running them individually allows them to pass. This is making it annoying to test changes locally. I have not been able to reproduce this on another machine since I don't have access to another one at the moment.

This is on a fairly beefy Intel NUC.

Thanks for reporting the issue! Lately I have the feeling no one is running the rpc tests anymore and travis is our only tester, which is suboptimal, as a segfault slipped through and existed for several weeks.

I think we should work to get any false positives out of the rpc test suite sooner rather than later.

Just a few points, some of which seem to have caused this Exception in the past:

• Are there any "zombie" regtest nodes running?
• Are there leftover folders/files in the tempdir you are using?
• Is there enough space in the tempdir and RAM for the fullblocktest?

Are there any "zombie" regtest nodes running?

running ps -aux | grep "bitcoind" leads me to believe not

Are there leftover folders/files in the tempdir you are using?

(assuming we're talking about /tmp) A bunch. I deleted them and nothing changed.

Is there enough space in the tempdir and RAM for the fullblocktest?

I have 32GB of RAM, and cleaning out tempdir didn't help.

Ok this is also occurring with 0.13.0, which I know used to work fine. Most likely a local-to-me regression.

There might be some hints in debug.log to indicate why bitcoind has failed to initialize. rpc-tests.py should output something like this for a failed test:

prioritise_transaction.py:
Initializing test directory /tmp/<test_directory>
INFO messages ...
Unexpected exception caught during testing: Exception('bitcoind exited with status 1 during initialization',)
Stopping nodes

and the temporary test directory shouldn't be cleared up.

Is there anything in <test_directory>/node<n>/regtest/debug.log?

2016-11-04 17:34:53.106184 Binding RPC on address ::1 port 16080 2016-11-04 17:34:53.106314 Binding RPC on address ::1 port 16080 failed. 2016-11-04 17:34:53.106327 Binding RPC on address 127.0.0.1 port 16080 2016-11-04 17:34:53.106352 Binding RPC on address 127.0.0.1 port 16080 failed. 2016-11-04 17:34:53.106361 Unable to bind any endpoint for RPC serve

Good call. For node0, no other nodes managed to start

And I see that port is taken up by another process.... let me investigate.

Yep, zombie daemons not named bitcoind. Closing.