Inconsistent coverage reports between builds with test retry enabled
We are using the dotnet runner to execute our NUnit tests and have it configured to calculate coverage using the built-in dotCover instance. However, we are inconsistently getting accurate code coverage reports.
Starting: "C:\Program Files\dotnet\dotnet.exe" C:\BuildAgent\tools\JetBrains.dotCover.CommandLineTools.2024.3.3\dotCover.dll cover C:\BuildAgent\temp\agentTmp\1dotCover.xml /ReturnTargetExitCode /AnalyzeTargetArguments=false21:16:39 Starting: C:\BuildAgent\tools\JetBrains.dotCover.CommandLineTools.2024.3.3\dotCover.dll cover C:\BuildAgent\temp\agentTmp\1dotCover.xml /ReturnTargetExitCode /AnalyzeTargetArguments=false21:16:39 Starting: .NET SDK 9.0.102 "C:\Program Files\dotnet\dotnet.exe" test C:\BuildAgent\work\7864ae8732345194\*****.sln --filter FullyQualifiedName!~UITests --configuration Release --no-build @C:\BuildAgent\temp\agentTmp\2.rsp --logger trx --results-directory TestResults21:16:39 in directory: C:\BuildAgent\work\7864ae8732345194
Because we submit code coverage reports to SonarQube cloud, I have a separate build step after the tests that use a powershell script to merge all the available .dcvr files and then convert them to html (the format SonarQube requires.) Frequently this step will fail because there's no *.dcvr file in the agent's temp directory. Other times it will succeed, but the coverage will be in the single digits, and other times, it works great and we get the 70% coverage that we expect.
echo 'Generating code coverage HTML report for Sonar Scanner'$files = ls %AgentTempDir%\*.dcvr -r | select -ExpandProperty Fullname$snapshot = [string]::Join(";", $script:files)
echo 'DotCover Merging files:' $files%SonarDotCover% merge /Source=$snapshot /Output=merged.dcvr
echo 'DotCover Generating Reports'%SonarDotCover% report /Source=merged.dcvr /Output=%SonarCoverageReport% /ReportType=HTML%SonarDotCover% report /Source=merged.dcvr /Output=dotCover.ndepend.xml /ReportType=NDependXML
*Note that %SonarDotCover% = %teamcity.tool.JetBrains.dotCover.CommandLineTools.DEFAULT%\dotCover.exe, and %AgentTempDir% is %system.teamcity.build.tempDir%\..\agentTmp\
I'm at a loss as to how to troubleshoot this, since dotCover is launching correctly, as is dotnet test.
I do have test retry set to 3 because we have a couple of flaky tests. Is it possible that dotCover is only looking at the last execution of dotnet test for some reason, which is why I occasionally get zero or very low coverage?
Is my problem that I'm manually merging the dotCover reports myself, and I should be using the dotCover runner to convert the reports to .html format? If so, how do I know what the output filename of the report is so that I can upload it to SonarQube cloud?
Please sign in to leave a comment.
Hi,
The problem you described seems to be caused by the following bug, fixed in 2025.07.1:
https://youtrack.jetbrains.com/issue/TW-95131
Can you check if upgrading to this version helps?