Fix zmq test flakiness #20934

There are many reports of the test being flaky: #20672 (comment)

Thus, it should be made more robust, as described in #20538 (comment)

Useful skills:

• Background in our functional test suite (python3)
• Background in zmq

Want to work on this issue?

For guidance on contributing, please read CONTRIBUTING.md before opening your pull request.

Of the last 571 failures, 22 are from the interface_zmq.py functional tests (3.8%). According to the numbers, it's the flakiest functional tests we have. @domob1812 @theStack @mruddy @n-thumann are any of you willing to give this a shot?

Took some time to look at the problem, it seems to be quite tricky to solve in a solid way. I tried the suggested method of "syncing up" via repeatedly generating a block and waiting for the expected message (until it doesn't timeout anymore), but generating a block seems to interfere with some of the sub-tests. It also already generates notification messages for our subs that are received later (even if we are not connected yet). Maybe something like this would work:

• restart node with additional pubhashtx test publisher (on a port not used by any of the test subs)
• repeatedly generate block and wait for expected messages from test publisher, until it doesn't time out anymore
• invalidate generated blocks
• clear mempool (needed?)
• read from our subscriber sockets until there is no data (a "reverse flush" so to say)

Maybe I'm thinking too complicated though. Whatever the solution will be, at least having a common test setup method should serve as a better basis for solving this issue: #20953

but generating a block seems to interfere with some of the sub-tests

Yes it would require making all the subtests more robust I think.

alternative setup

Seems pretty complicated, and with intentional block rollbacks things can get weird.

Could a mempool tx be used to sync up instead of a block?

What about temporarily disabling interface_zmq.py in CI until this is fixed?

It seems to me that interface_zmq.py as it is currently working is a net negative from a CI testing perspective due to its extreme flakiness :)

How often is it failing?

On Tue, Jan 26, 2021, 7:30 PM practicalswift notifications@github.com wrote:

What about temporarily disabling interface_zmq.py in CI until this is fixed?

It seems to me that interface_zmq.py as it is currently working is a net negative from a testing perspective due to its extreme flakiness :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/20934#issuecomment-767482690, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMAFU3JOQHWO4XZFSHAKP3S32RTTANCNFSM4WCTAKVQ .

Of the last 571 failures, 22 are from the interface_zmq.py functional tests (3.8%). According to the numbers, it's the flakiest functional tests we have.

(quote from @adamjonas )

interface_zmq.py flakiness is back and I think #21008 is hurting more than helping.

Before merge of #21008 on 2/16 (Feb 12-15): Failed 1 time on 1 PR (1,274 bullds)

Same Friday to Monday time period after merge (Feb 19-22): Failed 11 times across different 9 PRs (1,470 total builds)

Fixed in #21216 ?

ref #21310