Help Needed for Plugin to Improve Unit Test Coverage Statistics
My company currently uses Emma to generate statistics around unit test coverage. Currently we have so many lines of code that any individual checkin doesn't change the overall unit coverage stats. What I am looking for is the ability to cross reference the checkins diffs (lines added, updated, deleted), with the Emma line coverage stats to generate a version control commit statistic that displays: Unit Test Coverage % for added lines, Unit Test Coverage % for updated lines (obviously deleted lines won't be covered).
I think that the Unit Coverage % for newly added / updated lines is the most valuable metric for a development team. This simple metric will drive the correct behavior, which is to only commit unit tested code. If you update code and it wasn't previously tested, you should add unit tests before you commit the updates. Of course, I'd want to add historical reporting, and per user statistics later so you could have a leader board that showed the top 10 commiters to a project by unit test coverage.
So now for my question. Is there anything like this that currently exists for TeamCity or any other open source build server? If no solution currently exists, how hard would it be to write a TeamCity plug that after a build, looks at both the Emma output and the version control lines added/updated/deleted for each file and calculates the metrics that I listed above? I'm happy to write this plugin as I feel it would be helpful not only for our company but everyone following TDD (test driven development) or just seeking better commit level unit testing metrics. I'd like to make this an open source plugin because I strongly believe that these new metrics will help improve the overall quality of TDD and open source projects in general.
Again, developing this plugin is well within my skill set (7+ years Java development with extensive third party plugin experience), however from the TeamCity plugin docs, I don't know where to begin. So if this doesn't exist could someone tell me in TeamCity terms, what type of plugin this would be? How I can access both Emma and VC changes for a given build? Being a developer, I don't need extensive documentation, just pointing me at a couple open source plugins that are similar or have pieces that I need would be helpful. Basically, you write the words and I'll write the code.
Please help make the world of Unit Testing a better place!
Thanks in advance.
Please sign in to leave a comment.
I do not know whether such features exist in other continuous integration servers. Probably Atlassian Clover can provide similar statistics.
It's not that easy to obtain modified or added lines from version control (sometimes it can be really slow operation). Probably better approach is to compare coverage statistics from two builds (previous and current). Coverage statistics among other things contain classes with lines and status for each line. So it is possible to compute line coverage difference.
BTW TeamCity 5.0 will have one more coverage engine bundled: IntelliJ IDEA coverage. So far it works better than Emma and provides more accurate results. Also it applies on-the-fly instrumentation and does not modify classes on disk. But it does not have branch coverage yet.
Not so long ago I tried to generate diff report using coverage statistics for IDEA coverage engine. This report showed diffs in coverage per each class where coverage changed. But I had to postpone this feature because of more critical issues. I think something like this can also be done for Emma.