How do I make TeamCity break build n if metric x is lower that it was in build n-1
So I've got TeamCity humming along quite nicely. We (in a .NET shop) run tests, coverage, fxcop, ndepend (though only the default CQLs so far) during each build.
However, it's a mostly legacy code-base. So test-coverage is low-to-nothing (though growing). FxCop floods the inspections report with legions of terror and distastefulness. NDepend violations are a crushing horde.
Some projects are in better shape than others; we have a couple greenfield bits that we're keeping well tended. However, the legacy stuff presents a problem - don't want to turn on a draconian set of "Thou shalt ... <always use an IFormatProvider>" rules, since we'd grind to a halt making the build green again, and we'd be making changes that aren't covered by test-automation (and we have really gah-no-never-do-that-ever! things like hard-coding threads to particular locales and then expecting servers to be installed a particular way), and so we'd grind even more to a halt as we wrote test-automation for a system that's badly documented at best.
So, pragmatically, about the best _simplest_ thing we can do at at the moment is the keep-people-honest option, which is to fail the build if a checkin is detected that makes the code-base _worse_. Worse, as defined simplistically, like:
* Test-coverage goes down (blanket coverage; more interesting things like coverage within particular namespaces can wait)
* FxCop warnings increase
* FxCop errors increase
* NDepend CQL violations of rule x increase
* <Insert other broad-brush static analysis metric here> gets worse
How do I get TeamCity to do that for us? Obviously, not averse to tinkering with our build-script to do this.
One way that occurs to me is to get the build-script to publish said broad-brush metrics to a file and publish that at the end of a build (successful build or not), then as the last task of the build, pull the artifact from <.lastSuccessful>/artifacts (or whatever the URL would be), and do the arithmetic there.
Has anyone done this? How did you do it? Does it come out of the box?
Please sign in to leave a comment.
This is something we already think about. There is related issue in our tracker: http://youtrack.jetbrains.net/issue/TW-2539
Thanks - I've +1'd it.