fail build if test coverage less than master branch?

Answered

I'd like to fail feature branch builds if they have test coverage less than the last successful build of master


I tried adding a extra failure condition by following the steps here


https://www.jetbrains.com/help/teamcity/build-failure-conditions.html#BuildFailureConditions-AdditionalFailureConditions


But I can't find a way of referencing the last successful build of master


Is there a special tag automatically set to the "default" branch or do we need to add some code/build step to auto pin / auto tag it?

2
18 comments
Avatar
Fedor Rumyantsev

Hello Tony,

As of now, yes, you would need an extra step to automatically tag or pin a build. Two options I could come up with:
1) usage of REST API to add a tag or pin a build (https://www.jetbrains.com/help/teamcity/rest-api.html);
2) there is a third-party plugin (https://github.com/ocroquette/teamcity-autopin) which, per README, allows several options for auto-tagging/pinning builds; it appears that it has been written for 10.0 TeamCity version, but perchance it could be useful for your own implementation. 

We do have a feature request regarding service messages usage for tag updates (https://youtrack.jetbrains.com/issue/TW-14363), and I have just created another request regarding custom failure condition to compare a metric against default or custom branch (so to allow to compare feature branch with master): https://youtrack.jetbrains.com/issue/TW-65538

Please let me know if there are any details I could provide and I will gladly assist.

0

I've created a step to automatically tag the master/default branch builds. However, I'm getting this error from the code coverage metric failure condition:

Cannot find Latest build with tag: 'master', branch filter: 420/merge to calculate metric 'percentage of line coverage' for branch 420/merge

 

It seems that it is only looking for builds with the same branch as the current build.  How come this doesn't look at the default branch? How can I make it look at the default branch for the tag?

According to the documentation if no suitable build is found in the same branch, then it looks at the default branch:
https://www.jetbrains.com/help/teamcity/2019.2/working-with-feature-branches.html#WorkingwithFeatureBranches-FailureConditions

1

Hello Petar,

Am I correct to assume that you are using 2019.2? Please let me know the version; in the meantime I will try to reproduce the issue.

0

Yes, I am using TeamCity Enterprise 2019.2.4 (build 72059).

0

Hello Petar,

First of all, a word of notice - I have checked with 2020.1.5 release (as the logic around build failure branch finder was last changed in early 2019, well before the 2019.2.4 release). The issue could not be reproduced there - a minimal setup where a single build in default branch and a single build in feature branch is present, and configuration is designed to fail if branch test count is more by at least 1 test than the count for tagged branch, caused feature branch to fail and pushed correct metric change message into build log. Could you please send over:
1) a screenshot of Failure Condition setup?
2) a screenshot with builds in master branch, which shows status, started date and tag(s)?

0

Here's the failure condition:

Here's the master/default branch succeeded and tagged:

Here's the feature branch succeeded, but with the warning that it couldn't find a build to compare with:

0

Hello Petar,

Thank you; I have reproduced the issue on 2019.2.4, with exactly the same setup. Unfortunately, I could not find a related issue on our tracker, but given that the logic works on 2020.1.5, I would suggest to upgrade to latest version of TeamCity. In the meantime, I will check with the development team if there was a relevant fix between the versions and will get back to you.

Please let me know if upgrade helps with the issue.

0

Thanks for confirming! I'm not sure if we'll be able to upgrade soon, but it'll be helpful to know if there's working version closer to ours that might make upgrading easier.

EDIT: Looks like we'll be upgrading in a couple of weeks. I'll give an update then.

0

We just updated to TeamCity Enterprise 2020.1.5 (build 78938). Unfortunately we are getting the same exact warning as in the pictures before.

0

Hello Petar,

Sorry to hear that; just to double-check, if you remove the condition and set it up again, does it behave in the same fashion? 

0

Yes, disabling the failure condition, running, then enabling again, and running again still has the same issue.

0

Hello Petar,

I just have set up the environment for tests again:


The failure condition compares with last build with baseline tag; the only build with this tag is on main branch, but the build on screenshot is on a feature branch. The logic is properly detecting the last build.

Could you please check the DSL code for the build configuration in question as per below and share the build failure conditions section? I will dig through the relevant code for this feature in the meantime. 

0

The build configuration is based on a template so I will show you the template build configuration DSL for the failure condition, then the failure condition from the actual build configuration that I've been talking about in my previous posts.

I should not that one odd thing I noticed is that even though I deleted and re-created the failure condition on the template, it still shows 2019.2 for the version.

Here's the DSL from the template:

failOnMetricChange {
id = "BUILD_EXT_412"
metric = BuildFailureOnMetric.MetricType.COVERAGE_LINE_PERCENTAGE
units = BuildFailureOnMetric.MetricUnit.PERCENTS
comparison = BuildFailureOnMetric.MetricComparison.LESS
compareTo = build {
buildRule = lastSuccessful()
}
param("metricThreshold", "0.01")
}

Here's the DSL from the build configuration we've been talking about (inherited, and overriden):

failOnMetricChange {
id = "BUILD_EXT_412"
metric = BuildFailureOnMetric.MetricType.COVERAGE_LINE_PERCENTAGE
units = BuildFailureOnMetric.MetricUnit.PERCENTS
comparison = BuildFailureOnMetric.MetricComparison.LESS
compareTo = build {
buildRule = buildWithTag {
tag = "master"
}
}
param("metricThreshold", "0.01")
}

I hope that helps.

0

I just realized that I had made a mistake before. Yesterday I said that disabling and re-enabling didn't work. Today I actually deleted the failure condition, and recreated it. It seems using this method I am now getting the desired behavior:

Also, it seems that if there is no triggered build failure condition, then only the first line (warning) will be displayed. This makes it confusing because it doesn't tell you that it also checked against the master/main branch. Logging that info is something I recommend is added to future versions of TeamCity to avoid confusion.

Thanks a lot for your help!

0

Hello Petar,

Thank you for the notice, great to hear that the logic is working! The issue itself is weird; I was wondering if the saved configuration was somehow different from the UI representation, but the DSL you have shared before says otherwise. If you see the issue reoccur, could you please share the details with me so I will raise it to the development team?

Speaking of the default branch related warning - yes indeed, we show the condition-related marker (e.g. latest build, latest successful build), but not the branch where the build was found. I have reached out to the development team to discuss this further (e.g. if we could always add a line regarding the default branch is being checked as well) and will circle back to you when I have an update.

0

I'll keep an eye out if anything else stands out.

Great, that will be a nice improvement.

Thanks again!

0

Hello Petar,

Just a quick follow-up; I have created an issue on our tracker regarding the details on build selection logic in build log: https://youtrack.jetbrains.com/issue/TW-69005
Please feel free to vote/comment as you see fit.

0

Hello,

I might be having the same issue. TC Enterprise 2020.1.5 (build 78938), I am setting up this fail condition:

failOnMetricChange {
id = "BUILD_EXT_100"
metric = BuildFailureOnMetric.MetricType.COVERAGE_STATEMENT_PERCENTAGE
units = BuildFailureOnMetric.MetricUnit.PERCENTS
comparison = BuildFailureOnMetric.MetricComparison.LESS
compareTo = build {
buildRule = lastPinned()
}
param("metricThreshold", "0.01")
}

and the only builds this condition seems to pick up for comparison are from the same branch. I.e. dev builds seem to only compare to pinned build in dev branch, a build of feature-123 branch would only compare to a pinned build of feature-123 branch but not to pinned build of dev branch. 
The message I am getting is:

0

Please sign in to leave a comment.