Inconsistent test runs due to arbitrary "suite" naming

We are running dotcover and nunit3-console from a script buildstep.  (The reason we are not using the builtin nunit runner is that we want locally reproducible builds for devs.)

We are running into some issues with test name aliasing and test number inconsistencies due to a combination of the following factors:

  1. Some assembly names are quite long (~100 chars)
  2. The length limit on "full test name", i.e. the combination of "suite" + assembly name + test name
  3. From build to build, one of the test assemblies is chosen as the "suite" for the entire test run covering several assemblies, seemingly at random.

So, for some builds, one of the lengthy assembly names is somehow selected as the "suite" name.  If the resulting full name exceeds 255 chars, the result is that test names are truncated.  This leads to test aliasing, i.e. different tests are now detected as the same test run several times, because the truncated names are equal. As a result, reported test count and pass rates fluctuate.

Stripped down to the essentials, this is representative of the .bat script we use to run the tests:

%dotCover% cover dotcover_config.xml ^
/TargetExecutable="%nunit3_console%" ^
/TargetWorkingDir="." ^
/TargetArguments="%test_assemblies%" ^

The config file defines exclude filters for the test assemblies themselves.  The argument environment variable (%test_assemblies%) is a space-separated list of unit test assembly DLLs.

Following the run, we report results back to TC using a service message;

echo ##teamcity[importData type='dotNetCoverage' tool='dotcover' path='%dc_results%']

Given the apparent difficulty of addressing the 255 character limitation, we have tried specifying a suite name directly using "echo ##teamcity[testSuiteStarted name='UnitTests']" and a corresponding "testSuiteFinished" service message, but this seems to have no effect.

How can we control the suite naming for DotCover/TeamCity in a dependable way, so that we are able to achieve reproducible results between builds?  Shortening assembly names across the board in accordance with limitations in the coverage or build product would get us part of the way, but does not address the core issue.

Thanks in advance!



1 comment
Comment actions Permalink

Quick note on the run; it seems like the source of the "arbitrary suite" issue may be nunit3-console when running with multiple assemblies.  Looking into that.



Please sign in to leave a comment.