TC 4.5.5 Ignores category-include on NUnitLauncher resulting NUnit Category not being include in test run

A few days ago we upgraded from TC 3.1 to TC 4.5.5 and now our now our "slow" NUnit category tests are no longer running.  There have been no changes to our build script the only changes to the env has been the updated to TC 4.5.5.  The category-include ran fine in the prior version of TC.  After reviewing the  TC 4.5 documentation there does not appear to be any changes in calling syntax for NUnitLauncher.

We are leveraging NUnitLauncher to run our tests under NCover.  The first run leverages the category-exclude to only run our fast tests, the second uses category-include to run out slow category tests. We late merge the 2 NCover coverage files in another target.  This process was running fine but following the updated now the category-include is being ignored, no tests are run.

Example Unit Test:


[TestFixture]
[Category("slow"), Explicit]
 public class MyTestClass : ConnectedTest
{

     Test logic ...
}

Build Section to run Unit Tests:



<target name="UnitTestWithNCover" depends="PrepUnitTest,CreateCoverageDirs">
      <echo message="Start UnitTestWithNCover: "/>
      <tstamp verbose="true" />
     
      <!-- BEGIN Default run -->
      <property name="assembly.list.mod"  value="${assembly.list}" />
     
     <echo message="assembly.list.mod: ${assembly.list.mod}" /> 
      <echo message="Running with exclusions: ( /exclude:${unit.test.exclude} )"/>

      <CoverTests in.assemblies="${assembly.list.mod}"
                          in.command.line.exe="${teamcity.dotnet.nunitlauncher}"
                           in.command.line.args="v2.0 MSIL NUnit-2.4.7 /category-exclude:${unit.test.tc.exclude} ${BuildOutputPath}\${BuildTypeDirectory}\${assembly.test}"
                           in.coverage.file="${coverage.run1.file}"/>
      <!-- END Default run -->
 
     
     <echo message="Inbetween UnitTest, niteFlag=${flag.nightly}"/>
 

     <!-- BEGIN Full Nightly run slow tests -->
      <if test="${property::get-value('flag.nightly') == 'true'}">
           <echo message="Performing second run which includes: ( /include:${unit.test.nightly.include} ) "/>
 
           <property name="assembly.list.mod" value="${assembly.list}" />
     
          <echo message="assembly.list.mod: ${assembly.list.mod}" />

           <CoverTests in.assemblies="${assembly.list.mod}"
                               in.command.line.exe="${teamcity.dotnet.nunitlauncher}"
                               in.command.line.args="v2.0 MSIL NUnit-2.4.7 /category-include:${unit.test.nightly.include} ${BuildOutputPath}\${BuildTypeDirectory}\${assembly.test}"
                               in.coverage.file="${coverage.run2.file}"/>
      </if>
      <!-- END Full Nightly run slow tests -->
 
     
     <!-- Prepare TC Artifacts -->
      <call target="TCArtifacts" />
 
     
     <echo message="End UnitTestWithNCover: "/>
      <tstamp verbose="true" />
</target>

13 comments
Comment actions Permalink

In 4.5 we added support for [Explicit] NUnit tests. According to the documentation, such tests can only be started if the category is specified. Please consider removing Explicit attribute, or add the category name to the NUnit runner settings.

0
Comment actions Permalink

Okay.  Here is the exact bug we are experiencing - http://youtrack.jetbrains.net/issue/TW-9168

We are currently running TC 4.5.5 build 9103 which is stated to have been fixed in the issue item above.  However, we are still experiencing the above described issue.  In our case we have defined the Explicit at the TestFixture Category level not the individual test level but either method should work. Nothing in our build script has changed, just upgrade from TC 3.1 to 4.5.5.


We are adding the NUnit Category to the category-include parameter in the NUnitLauncher call.  So, we call target CoverTests below.  In CoverTests we are using the NCover task to call the NUnitLauncher to run our NUnit tests.  It is important to note we define the profiled process module as JetBrains.TeamCity.PlatformProcessRunner.msil.exe.  The Explicit Category is called "slow", which is defined in the
${unit.test.nightly.include}.

${unit.test.nightly.include}  = "slow"



  <CoverTests in.assemblies="${assembly.list.mod}"
                              in.command.line.exe="${teamcity.dotnet.nunitlauncher}"
                              in.command.line.args="v2.0 MSIL NUnit-2.4.7 /category-include:${unit.test.nightly.include} ${BuildOutputPath}\${BuildTypeDirectory}\${assembly.test}"
                              in.coverage.file="${coverage.run2.file}"/>


****************************************************


<macrodef name="CoverTests">
      <attributes>
           <attribute name="in.assemblies" />
          <attribute name="in.command.line.exe" />
           <attribute name="in.command.line.args" />
           <attribute name="in.coverage.file" />
      </attributes>

     <sequential>

           <!-- BEGIN Default coverage -->
           <echo message="BEGIN unit tests and coverage xml." /> 
           <ncover
                     program="${path.ncover.exe}"
                     commandLineExe="${in.command.line.exe}"
                     commandLineArgs="${in.command.line.args}"
                    profiledProcessModule="JetBrains.TeamCity.PlatformProcessRunner.msil.exe"
                    coverageFile="${path.report.dir}\${in.coverage.file}"
                   logLevel="Verbose"
                     logFile="${path.report.dir}\${coverage.log.file}"
                     verbose="true"
                     assemblyList="${in.assemblies}"
           /> 
           <echo message="COMPLETED unit tests and coverage xml." /> 
           <!-- END Default coverage -->

 
           <if test="${property::get-value('flag.nightly') == 'false'}">
           <!--
                BEGIN report generate for single run
 
                ONLY generate the report when a nightly is run
                NOT a full nightly.
             --> 
           <echo message="BEGIN report coverage creation." />
 
           <ncoverexplorer
                program="${path.ncover.explorer.exe}"
                projectName="ACS"
                reportType="ModuleClassFunctionSummary"
                outputDir="${path.report.dir}"
                xmlReportName="${report.file}.xml"
                htmlReportName="${report.file}.html"
                showExcluded="True"
                verbose="True"
                satisfactoryCoverage="14"
                failCombinedMinimum="true"
                minimumCoverage="2.0">
                <fileset>
                       <include name="${path.report.dir}\${in.coverage.file}" />
                </fileset>
                <exclusions>
                       <exclusion type="Assembly" pattern="*.Tests" />
                       <exclusion type="Namespace" pattern="*.Tests*" />
                </exclusions>
           </ncoverexplorer> 
 
           <echo message="COMPLETED report coverage creation." />
           <!-- END report generate for single run -->
     </if>
 
 </sequential>
 </macrodef>

0
Comment actions Permalink

Will the following test will work:

[TestFixture]
public class Test {

  [Test, Explicit, Category("slow")
  public void test_slow() { .. .}
}

0
Comment actions Permalink

I will try this and report back to the thread.  Thanks.

0
Comment actions Permalink

We have verified that neither unit tests marked as explicit at the class or function level in separate classes is run during our build.


Class Level

[Explicit, Category("slow")]
public class CaseActivityLogItemListTest : ConnectedTest




Functional Level

[Test, Explicit, Category("slow")]
  public void CompareAppDBVersions()

0
Comment actions Permalink

Please try TeamCity 5.0 EAP. The issue was exactly fixed for it.

0
Comment actions Permalink

According to the documentation in this issue - http://youtrack.jetbrains.net/issue/TW-9168 what we are experiencing has been resolved in TC 4.5.5 build 9103 and EAP 5.0.  Are you saying this has only been addressed in EAP 5.0?  If we upgrade to EAP 5.0 will we be able to upgrade to 5.0 release?  Do you know when EAP 5.0 will be released?

0
Comment actions Permalink

I have reproduced the issue in TC 4.5.5. I'll send you a patch.

0
Comment actions Permalink

Okay.  I appreciate the time and look forward to receiving the patch.

Sean

0
Comment actions Permalink

Is that acceptable for you to try TeamCity 5.0 EAP build?

I've attached a patched version of build Agent .NET part. To apply the patch, please do the following:
- open <server>/webapps/ROOT/update/plugins
- backup dotnetPlugin.zip
- replace it with attached dotnetPlugin-9137p.zip
- wait all build agents to update

Thanks!



Attachment(s):
dotnetPlugin-9137p.zip
0
Comment actions Permalink

Ok, I have updated the server ( for Sean )  to  5.0 EAP RC1

Put the new plugin zip in the update dir and indeed the agents updated automatically, very slick.

Once that's done I do not have any compatible agents anymore, see below, please advice.

Thanks,
  Orn

tc.jpg

0
Comment actions Permalink

I mean to try TeamCity 5.0 or to patch TeamCity 4.5.x
If you downloaded TeamCity 5.0, please revert the patch to try the fix.

0
Comment actions Permalink

Yes yes it works now, thanks a lot.

Orn

0

Please sign in to leave a comment.