Flaky tests high flip rate remains unchanged after more than a week
Answered
Hi,
We have several tests that were indeed flaky around two weeks ago.
In one example I checked, the last failure was 12 days ago but the flaky test detector still claims that the test has frequent test status changes.
At the same time the test was fixed, the TeamCity env. changed and we have new hosts for all the agents.
Could it be that the flip rate measured per agent does not have a time window? Does the flip rate per agent have a time window different than the week (mentioned at the top of the flaky tests tab)?
Is there a way to reset the flaky test detector time window?
Thanks
Please sign in to leave a comment.
Hello,
Normally, outdated test failures (i. e. those which have left the sliding window) will be cleaned up when server clean-up starts. If you have clean-up scheduled to run earlier than once a week, or have it disabled, outdated failures will still contribute to the flaky statistics.
You may wish to set the teamcity.flakyTestDetector.cleanupOnBuildCompletion internal property to true in order to run failure statistics clean-up once per build (i. e. whenever a build finishes).
OK. A restart of the TeamCity server got rid of all flaky tests which last failed more than a week ago.
This suggests a bug in the detector.