tests: Some issue about running functional tests on Windows #14446

Since #14007, the functional tests would run on Appveyor CI. However the tests would fail occassionally due to several reasons. Here is the list:

• ConnectionAbortedError: [WinError 10053] might be a race condition Maybe fixed by #14670 For reference:
  https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19366470 https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19344806

• PermissionError: [WinError 32]: found in feature_notifications. The test script delete the files before the notification command close the file. Maybe fixed by #14465 For reference: https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19363071 https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19342222

• OSError: [WinError 10022] An invalid argument was supplied: Unknown reason. One-off? For reference: https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19365696

• AssertionError: Block sync timed out: Unknown reason. Found in wallet_txn_doublespend For reference: Maybe fixed by #15419 https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19360284 https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19334394 https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/20031316

• PermissionError: [Errno 13] Permission denied: Unknown reason. Found in get_auth_cookie function Maybe fixed by #14788 For reference: https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19324004 https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19393533

If you find any fail that is not listed here. Please comment below.

Found a new one? in feature_uacomment.py self.nodes[0].assert_start_raises_init_error(["-uacomment=" + unsafe_char], expected, match=ErrorMatch.FULL_REGEX)

https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19493806

Found a new one? in feature_uacomment.py self.nodes[0].assert_start_raises_init_error(["-uacomment=" + unsafe_char], expected, match=ErrorMatch.FULL_REGEX)

https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19493806

That is the first situation, you can see ConnectionAbortedError: [WinError 10053] in logs.

This is pretty annoying, as it adds a big red failure sign to your PR. Can we disable appveyor, or prevent it from influencing the CI status of PRs?

I'm still seeing ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine, for example on https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/19954225.

@sipa #13501 isn't merged yet.

@ken2812221 #14670 is merged, can you update this?

I still see the block sync timed out issues.

Also there is a failure in wallet_multiwallet, which might happen on travis as well. https://ci.appveyor.com/project/DrahtBot/bitcoin/builds/20846360?fullLog=true#L85

test_framework.authproxy.JSONRPCException: Wallet file verification failed: Error loading wallet w3. Duplicate -wallet filename specified. (-4)

Maybe this should be closed and individual issues should be filed for each failure reason?

If the issues persist, we might want to ignore the exit code for functional tests for now. That way, the output can still be checked manually, but a failing test wouldn't mark a pull request red in the pull request overview tab.

Maybe that diff would work:

diff --git a/.appveyor.yml b/.appveyor.yml
index eb2d32e1bc..492147a222 100644
--- a/.appveyor.yml
+++ b/.appveyor.yml
@@ -54,5 +54,5 @@ test_script:
 - cmd: src\bench_bitcoin.exe -evals=1 -scaling=0 > NUL
 - ps:  python test\util\bitcoin-util-test.py
 - cmd: python test\util\rpcauth-test.py
-- cmd: python test\functional\test_runner.py --ci --force --quiet --combinedlogslen=4000 --failfast
+- cmd: python test\functional\test_runner.py --ci --force --quiet --combinedlogslen=4000 --failfast || echo "ignore test failure"
 deploy: off

FYI I tracked down the wallet_txn_doublespend.py test failure -- it's due to an actual bug in the test. More explanation in the OP here: #15360.

The underlying problem is a race condition between connecting two nodes and generating a block on one of those nodes to trigger block sync on the other. I thought in #15360 that the best way to fix this would be to change the properties of the cached datadir that we create in our test framework, but it seems to me now that probably the test should be modified/rewritten so that the nodes start off outside of IBD, which would eliminate the race condition on block sync issue altogether.