test_framework: detect failure of bitcoind startup #7744

Replace the bitcoin-cli -rpcwait after spawning bitcoind with our own loop that detects when bitcoind exits prematurely (before RPC is up).

This prevents a hang in such a case (see #7463).

Not ready to merge yet. I've tested this with a -salvagewallet loop:

Unexpected exception caught during testing: bitcoind exited with status 1 during initialization
  File "/data/src/bitcoin/qa/rpc-tests/test_framework/test_framework.py", line 135, in main
    self.run_test()
  File "./wallet.py", line 295, in run_test
    self.nodes = start_nodes(3, self.options.tmpdir, [[m]] * 3)
  File "/data/src/bitcoin/qa/rpc-tests/test_framework/util.py", line 283, in start_nodes
    return [ start_node(i, dirname, extra_args[i], rpchost, binary=binary[i]) for i in range(num_nodes) ]
  File "/data/src/bitcoin/qa/rpc-tests/test_framework/util.py", line 267, in start_node
    wait_for_bitcoind_start(bitcoind_processes[i], url, i)
  File "/data/src/bitcoin/qa/rpc-tests/test_framework/util.py", line 144, in wait_for_bitcoind_start
    raise Exception('bitcoind exited with status %i during initialization' % process.returncode)
Stopping nodes

Even though reporting is better, it still hangs after "Stopping nodes", going to investigate why.

Should be fixed now. The problem was that start_nodes can fail with part of the nodes already started. wait_bitcoinds will subsequently wait forever for them to exit (as it uses a global variable to communicate bitcoind_processes). One-to-last commit makes sure that they're stopped properly in that case.

This is faster than the current -rpcwait construction, so #7745 triggers more, seems I need to solve that too here. Thanks Marco Falke for fixing this already.

Good idea! utACK 018b60c5ea703ed12edcde034a185f79e77e5576

utACK 018b60c