Count test as successful if at least one try is successful

Answered

We have added retries to our Selenium tests because they sometimes fails due to several reasons but when the test is retried it succeeds. We want to handle those as a success instead of a failure in our build process. TeamCity shows how many times it failed and succeeded. For example "The test was run 2 times in the build, 1 failure". 

Is it possible to say that this test is OK in TeamCity? It will then of course fail if none of the retries is a success. 

 

TeamCity Professional 9.0 (build 32060)

1 comment

Hello Martin,

Most probably we'll not implement a logic when a successful test run in the same build will fix the previous failure.

This will contradict with the current behaviour when invocationCount is set to detect flacky tests. In this situation, any failure means that the test fails.

If you have such tests, you should mute them so they don't break your builds, and make them more stable afterwards. Would it work?

Also in the upcoming TeamCity 10.0 version we add Fluky test detection feature.

0

Please sign in to leave a comment.