TestNG tests result custom rules

Hi guys,

I need to customize TestNG reporter to report a SUCCESS for tests that are retried multiple times on failure but eventually passes. For example: Test1 attempt#1 fails, but attempt#2 succeeds. This should be considered a success.

To achieve this effect, I have been experimenting with gradle TestNGOptions by turning off default listeners (useDefaultListeners=false) add a custom implementation of IReporter that scrubs its tests results data, but it seems TeamCity uses a custom IReporter implementation to report on TestNG results that it always reports a FAILURE on any failed test retry attempts.

I also don't see any option available in the TC config that would allow something like this.

Alternatively, a solution that would allow TC to show a custom build state (an orange icon instead of a successful green or a fail red) when there exist tests by the same name that fails and succeeds. However, I'm not sure TeamCity supports custom conditional rules for this use case.

Thanks in advance for any advice!

Hussein

1 comment
Comment actions Permalink

Hi Hussein,

The logic when successful test run in the same build fixes the previous failure will contradict with the current behavior when invocationCount is set to detect flacky tests. In this situation, any failure means that the test fails.
I think that if you have such tests, you should mute them so they don't break your builds, and make them more stable afterwards. Would it work?
Otherwise please feel free to create a feature request in our tracker.

0

Please sign in to leave a comment.