TeamCity "Lines" code coverage summary value does not match the NCover symbol coverage value

I've got the NCover code coverage working against our solution and I'm now getting the extra code coverage statistics on the "Overview" tab for each build.  The "Overview" shows the code coverage summary values for "Classes", "Methods", "Blocks" and "Lines".

I'm also producing the NCover full coverage report using the standard //or FullCoverageReport:Html:{teamcity.report.path} argument.

When I compare the code coverage summary values from the "Overview" tab to the summary values in the NCover full coverage report 3 of the 4 values match exactly ("Classes", "Methods" and "Blocks").  However, the Team City value for "Lines" does not match the NCover value for "Symbol Coverage".  I was expecting it to match just like the other numbers do.

See the attached (cut-and-pasted) screen shot which shows the two different values highlighted next to each other.

Interestingly the build log shows both numbers.  The log for NCover shows the value that it reports in the full coverage report, but there is also a team city log which reports the value(s) shown on the Team City "Overview" tab.  

Here's the relevant extracts from the build log (I've truncated parts of it for clarity):

[09:12:31]:  [Step 2/2] NCover Complete v3.4.16.6924 x86 (Administrator)
...
[09:12:34]:  [Step 2/2] Running NUnit-2.5.9 tests under .NET Framework v2.0 x86

...
[09:12:49]:  [Step 2/2] ***************** End Program Output *****************
[09:12:49]:  [Step 2/2] Execution Time: 17.7458 s
[09:12:49]:  [Step 2/2] Symbol Coverage: 28.78% <== This value is not the same as ...
...
[09:12:55]: Generate Coverage statistic values
[09:12:57]:  [Generate Coverage statistic values] Started NCover.Reporting tool: ...
[09:12:57]:  [Generate Coverage statistic values] Output: NCover Reporting Complete v3.4.16.6924 x86 (Administrator)
...
[09:12:57]:  [Generate Coverage statistic values] NCover.Reporting tool exited with code: 0
[09:12:57]: ##teamcity[buildStatisticValue key='CodeCoverageL' value='26.12921']  <== ... this value
[09:12:57]: ##teamcity[buildStatisticValue key='CodeCoverageAbsLCovered' value='1909.0']
[09:12:57]: ##teamcity[buildStatisticValue key='CodeCoverageAbsLTotal' value='7306.0']

Is this expected behaviour, or is this a bug?

We're running Team City Enterprise 6.0.2 with NCover 3.4.16.6924

Many thanks,
David



Attachment(s):
TeamCityNCover issue.jpg
5 comments
Comment actions Permalink

TeamCity uses vl and ul attributes from trendcoveragedata/stats element or report xml file that generated from NCover3 reporter with '//or MethodModuleNamespaceClassMethod:Xml' mode. Could you please attach (or email to teamcity-feedback@jetbrains.com) sample NCover3 report file where TeamCity fetches wrong stats.

0
Comment actions Permalink

Aha - got it!  I see the extra task in the build log for "Generate Coverage statistic values" that uses the command line you mentioned.

I grabbed the report.xml file produced on the build server and it has the following:

<?xml version="1.0" encoding="utf-8"?>

<trendcoveragedata>
  <en>::View::</en>
  <stats acp="95" afp="80" abp="95" acc="20" ccavg="1.3" ccmax="13" ex="0" ei="1" ubp="2127" ul="5397" um="1512" usp="3897" vbp="1096" vl="1909" vsp="1575" mvc="229" vc="1" vm="889" svc="10658" />


Looking at the "ul" and "vl" attributes (in blue), these match exactly what is being reported on the "Overview" tab.    The values that are reported on the Full Coverage Report are the "usp" and "vsp" attributes (in red).

So my conclusion is that this is because of a geniune difference that NCover is reporting between "symbols" and "lines".  The number of "symbols" covered is genuinely less than the number of "lines" covered.

I suspect that this is because you can have 1 "symbol" (i.e. 1 executable statement) that can be split over several "lines".  For example, Linq queries are typically split over multiple lines to aid readability even though they are only 1 statement.

Therefore, I don't think it's a bug, it's just an inconsistency.  The "Overview" tab and the Full Coverage Report are genuinely reporting different things.  It would be nice if they were both the same (just like the other 3 values)!

Thanks,
David
0
Comment actions Permalink

Thank you for information and description.

In TeamCity we also have statement coverage value that is possible to be added. Is the symbol coverage is equal to statement coverage? What is the difference in notions? I bevieve it would be easy to report statement coverage for this example too.

0
Comment actions Permalink

Yes I believe symbols and statements would be the same thing.

If you look at the NCover definition of "Code Quality Metrics" (http://docs.ncover.com/best-practices/code-quality-metrics/) there are 4 main metrics defined:

  • Symbol Coverage
  • Method Coverage
  • Branch Coverage
  • Cyclomatic Complexity


These 4 metrics are part of the summary you get in a Full Coverage Report (see my previous screenshot).  It would be great if these metrics were automatically displayed and tracked by Team City on the "Overview" tab.
0
Comment actions Permalink

I've added an issue to the issue tracker for that. Please vote for it to be notified on any changes.
http://youtrack.jetbrains.net/issue/TW-17048

0

Please sign in to leave a comment.