Inconsistent coverage reports between builds with test retry enabled

We are using the dotnet runner to execute our NUnit tests and have it configured to calculate coverage using the built-in dotCover instance. However, we are inconsistently getting accurate code coverage reports.

Starting: "C:\Program Files\dotnet\dotnet.exe" C:\BuildAgent\tools\JetBrains.dotCover.CommandLineTools.2024.3.3\dotCover.dll cover C:\BuildAgent\temp\agentTmp\1dotCover.xml /ReturnTargetExitCode /AnalyzeTargetArguments=false
21:16:39     Starting: C:\BuildAgent\tools\JetBrains.dotCover.CommandLineTools.2024.3.3\dotCover.dll cover C:\BuildAgent\temp\agentTmp\1dotCover.xml /ReturnTargetExitCode /AnalyzeTargetArguments=false
21:16:39     Starting: .NET SDK 9.0.102 "C:\Program Files\dotnet\dotnet.exe" test C:\BuildAgent\work\7864ae8732345194\*****.sln --filter FullyQualifiedName!~UITests --configuration Release --no-build @C:\BuildAgent\temp\agentTmp\2.rsp --logger trx --results-directory TestResults
21:16:39     in directory: C:\BuildAgent\work\7864ae8732345194

Because we submit code coverage reports to SonarQube cloud, I have a separate build step after the tests that use a powershell script to merge all the available .dcvr files and then convert them to html (the format SonarQube requires.) Frequently this step will fail because there's no *.dcvr file in the agent's temp directory. Other times it will succeed, but the coverage will be in the single digits, and other times, it works great and we get the 70% coverage that we expect.

echo 'Generating code coverage HTML report for Sonar Scanner'
$files = ls %AgentTempDir%\*.dcvr -r | select -ExpandProperty Fullname
$snapshot = [string]::Join(";", $script:files)

echo 'DotCover Merging files:' $files
%SonarDotCover% merge /Source=$snapshot /Output=merged.dcvr

echo 'DotCover Generating Reports'
%SonarDotCover% report /Source=merged.dcvr /Output=%SonarCoverageReport% /ReportType=HTML
%SonarDotCover% report /Source=merged.dcvr /Output=dotCover.ndepend.xml /ReportType=NDependXML

*Note that %SonarDotCover% = %teamcity.tool.JetBrains.dotCover.CommandLineTools.DEFAULT%\dotCover.exe, and %AgentTempDir% is %system.teamcity.build.tempDir%\..\agentTmp\

I'm at a loss as to how to troubleshoot this, since dotCover is launching correctly, as is dotnet test.

I do have test retry set to 3 because we have a couple of flaky tests. Is it possible that dotCover is only looking at the last execution of dotnet test for some reason, which is why I occasionally get zero or very low coverage? 

Is my problem that I'm manually merging the dotCover reports myself, and I should be using the dotCover runner to convert the reports to .html format? If so, how do I know what the output filename of the report is so that I can upload it to SonarQube cloud?

0
1 comment

Hi,

The problem you described seems to be caused by the following bug, fixed in 2025.07.1:
https://youtrack.jetbrains.com/issue/TW-95131

Can you check if upgrading to this version helps?


 

0

Please sign in to leave a comment.