Allow tests to pass even when stderr got populated #10241

pull jonasschnelli wants to merge 1 commits into bitcoin:master from jonasschnelli:2017/04/test_stderr changing 1 files +5 −0

jonasschnelli commented at 6:57 PM on April 20, 2017: contributor

No description provided.
jonasschnelli added the label Tests on Apr 20, 2017
laanwj commented at 8:54 AM on April 21, 2017: member

I thought about this a bit: apparently some people use the stderr output to determine whether a test emitted e.g. sanitizer warnings. Possible idea: add a "WARNINGS" status, that is interpreted as PASS for the final tally but does show that something unexpected was printed.
Allow tests to pass even when stderr got populated, flag them as "passed with warnings" 41f392cf50
jonasschnelli force-pushed on Apr 29, 2017
jonasschnelli commented at 2:39 PM on April 29, 2017: contributor

Added a new status "Passed with warnings" which gets set when the test passed but stderr was not empty. The test will be flagged as "successful" for that new status.

in test/functional/test_runner.py:413 in 41f392cf50

 408 | @@ -407,6 +409,9 @@ def __repr__(self):
 409 |          if self.status == "Passed":
 410 |              color = BLUE
 411 |              glyph = TICK
 412 | +        if self.status == "Passed with warnings":
 413 | +            color = ORANGE

MarcoFalke commented at 10:14 AM on April 30, 2017:

NameError: name 'ORANGE' is not defined

jnewbery commented at 1:29 PM on May 1, 2017:

ORANGE would need to be a constant defined at the top of this file (see around L47)

in test/functional/test_runner.py:412 in 41f392cf50

 408 | @@ -407,6 +409,9 @@ def __repr__(self):
 409 |          if self.status == "Passed":
 410 |              color = BLUE
 411 |              glyph = TICK
 412 | +        if self.status == "Passed with warnings":

jnewbery commented at 1:29 PM on May 1, 2017:

elif please

jnewbery commented at 1:35 PM on May 1, 2017: member

Couple of comments inline.

If you add a new status string, you'll need to update print justification (otherwise I think your new category will just be truncated). Justification is on L337 in print_results() and L422 in TestResult.__repr__().

Now that we have 4 categories, it may make sense to define all the categories as constants at the top of the file and have a new MAX_STATUS_WIDTH constant as well.

One you've done that, can you add a screenshot of what this looks like?
MarcoFalke commented at 10:45 AM on May 14, 2017: member

Concept ACK, but I will close this pull in case no one picks it up in the next couple of days.
MarcoFalke commented at 11:49 PM on May 18, 2017: member

Closing for now. Feel free to pick this up again.
MarcoFalke closed this on May 18, 2017
jnewbery cross-referenced this on Jun 29, 2017 from issue [tests] Allow tests to pass when stderr is non-empty by jnewbery
MarcoFalke referenced this in commit f1f1605c22 on Jul 25, 2017
bitcoin locked this on Sep 8, 2021

Contributors

Labels

Linked (view graph)

#10703 [tests] Allow tests to pass when stderr is non-empty