Anyway to change the Flaky tests analyser settings?

Are there any settings, at the server or configuration/project level, that change the behavior of the Flaky tests analyser? I could not find any documentation and the plugin code seems to indicate al settings are hardcoded.

6 comments
Comment actions Permalink

Hi Hans,

could you be explicit as to what you would like to change? At the moment it's pretty rigid, but we can check whether there are any possibilities, either present or future if you are more specific.

0
Comment actions Permalink

Hi Denis,

Some of our TC users ask me if they could reset or fine tune flaky test detection, as it was not working from them (read the feature was not performing as expected). As I have server admin access, and they don't, I had a look if at the values could be reset or the detection tuned, but from what I found online that doesn't seems to be the case. Hence my question to get this confirmed.

I don't know the details of their setup, so it is hard for me to comment on how it is not exactly working for my colleagues.

If I look at the flaky test detection for the projects I'm involved with I also notice too much incorrect detections, mainly because it seems to look at all failing runs of a tests, even out-side of the scope of the project... we have a setup where we build a project and test it, we test sub-components of the project, some still in development, so ok to fail, others in release state. We (re)use the same test to test the different sub-components, organized in TC in different test configurations, but still the flaky test detection seems to report the flip rate based on the results of all sub-component test runs.

Example: we have test foo, for sub-project A, B and C. A should not fail as we plan to release it, For B the test should work, but fails from time to time, so the test is not muted, but it is also not a priority to get it fixed at this point in time (component is still in development). For C, which is also in development it fails 100% of the time, it is muted.

  • Flaky test of TC sub-project A shows that foo has 17 failures, with as flip rate of 18%. The test history of A alone is 92% (94 runs/7 fails)...
  • Flaky test of TC sub-project B shows that foo has 17 failures, with as flip rate of 18%. The test history of B alone is 63% (27 runs/10 fails)...
  • Flaky test of TC sub-project C shows that foo has 17 failures, with as flip rate of 18% and that the test is muted. The test history of A alone is 0% (23 runs/23 fails)...

For C it would have been nice to have had the option discard muted tests from the list (by default).

I assume the results of A and B get summed, but the are independent as the are in separate sub-projects. That the get summed for the parent project is ok, but not for the sub-project as it yields the check useless to us.

1
Comment actions Permalink

We are experiencing a similar setup.

Also, we have tests running in cloud-setups, which sometimes fails due to environment conditions.

We would like to be able to "trust" the flaky-identification again, but we cannot find a good way to reset the flaky detection. I.e. "reset flaky history and keep an eye on the future only".

0
Comment actions Permalink

Hi all, sorry this fell through the cracks. There are some options using internal.properties, basically:

(a) change the size of the sliding window (via teamcity.flakyTestDetector.timeFrame.days, the default is 7).
(b) disable some of the heuristics ("time frame", via teamcity.flakyTestDetector.enabled.heuristics, the default is DifferentStatusAtSameVcsChangeHeuristic,FlakyFailureHeuristic,FlippedInBuildWithoutChangesHeuristic,FlipRateTooHighHeuristic)

Beyond that, please forward any suggestions to our issue tracker: https://youtrack.jetbrains.com/issues/TW

0
Comment actions Permalink

Denis I would like to make use of the internal.properties you have described - can you tell me how/where these need to be configured?

0

Please sign in to leave a comment.