Inconsistent Behavior on Failure Condition: Number of tests decrease
Hi there.
We are currently experiencing some strange behavior on our TeamCity environment with the "Failure Condition" - "Fail build on metric change - Fail if number of tests is 5% less than the anchor value".
We have multiple PRs. For some branches TC reports a test failure due to a 14% drop of tests which is right (pull/40). But for pull/23 which has the same drop, the tests are green. This seems quite unlogical to me, shouldn't the pull/23 fail too.
Actually I would prefer to let any test of PRs fail, if the test number decreases in comparison to the last successful build on the main branch (develop). This way we can catch PRs which would wrongly decrease the number of tests.
Does somebody know how it can come to such a situation? Is this an expected (special) behavior or simply a bug. The pull request branches are organized like this:
pull/23 is branched off develop and the PR will merge back to develop. (main feature)
pull/40 is branched off pull/23 and will merge back into pull/23. (sub-extension of feature)
EDIT: we're on TeamCity Enterprise 2020.1 (build 78475)
Greetings
Daniel
Please sign in to leave a comment.
Hello Daniel,
As per this article: https://www.jetbrains.com/help/teamcity/working-with-feature-branches.html#Failure+Conditions, we will compare the changes in metric value with the build from the same branch (provided that a suitable build is found) or from the default branch as a fallback option - only if a suitable build was not found. In the above example, it appears that during the execution of #466 the last successful build found was #461, which had the same test count so the failure condition did not trigger.
I hope this helps; please let me know if there is anything else I could assist with.
Awesome, thanks for the insights there. I could confirm that there was a successful build on pull/40 with 1138, after that all tests failed due to the test number decrease.
Looks like everything is as expected according to the current TC behavior. I might check if there is already a feature request in YouTrack which would tackle the "fail if main branch has less tests" scenario. Would be nice to let PRs fail if somebody reduces the test count significantly on their branches.
Thanks again, topic closed 😃
Daniel,
You are most welcome! Speaking of the feature requests - yes, there is one: https://youtrack.jetbrains.com/issue/TW-61205 (the scope is to allow to compare metric against builds from a specific branch - in your case, you could set it to compare against develop for the desired effect). Please feel free to vote/comment on the feature as you see fit.