Recently failed tests do not run first

I use the MSBuild runner to run NUnit tests. The 'Run recently failed tests first' feature is ticked but the tests get no reordered.

I'm using TeamCity 4.0.2 build 8222

TeamCity configuration:

Build runner: MSBuild

Build file path: Project.msbuild

MSBuild version: Microsoft .NET Framework 3.5

MSBuild tools version: 3.5

Run Platform: x86 (Agents running under xp)

Targets: UnitTests

Command line parameters: /p:TestCategory=teamcity /p:TCNunitLauncher=%system.teamcity.dotnet.nunitlauncher%

Run recently failed tests first is ticked


MSBuild script:

<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" ToolsVersion="3.5">
  <PropertyGroup>
        <NUnitAssemblies>bin\Debug\Company.Product.core.NUnit.dll</NUnitAssemblies>
  </PropertyGroup>
 
  <Target Name="UnitTests">
    <Exec Command="$(TCNunitLauncher) ANY x86 NUnit-2.4.8 /category-include:$(TestCategory) $(NUnitAssemblies)" />
  </Target> 
</Project>


Tried to directly call the nunit-console runner once. Also this did not work. For this i changed the Exec Command in the MSBuild script to the following:

Command="vendor\NUnit\nunit-console.exe $(NUnitAssemblies) /noshadow /include=$(TestCategory) /config=Debug"
The nunit-console.exe is checked-in in our repository.

All tests with category 'teamcity' are executed and i can see the on-the-fly tests reporting but the tests run always in the same order.

What am i doing wrong?

12 comments
Comment actions Permalink

TeamCity does not support tests reordering when TeamCity addin for NUnit is used. To make tests reoreding work, you need to
use TeamCity embedded NUnit runner.

There are 2 possibilitues:
- use teamcity nunit runner process: http://www.jetbrains.net/confluence/display/TCD4/TeamCity+NUnit+Test+Launcher
- use teamcity MSBuild <NUnit> task: http://www.jetbrains.net/confluence/display/TCD4/NUnit+for+MSBuild

To check your script is running inder TeamCity you may add the following condition in MSBuild:
   Condition=" '$(TEAMCITY_VERSION)' != '' "

Thanks!

0
Comment actions Permalink

I am using %system.teamcity.dotnet.nunitlauncher% in the first above described example. It also runs directly within a command line runner, but no reordering (i know there is no 'run recently ...' checkbox).
Have still no clue why the MSBuild does not work with the TeamCity dotnet nunitlauncher. Is it because of the .NET 3.5?

0
Comment actions Permalink

TeamCity nunit runner is written on .NET 1.1. You may select what .NET Framework to use to run your tests: .NET 1.1 or .NET 2.0
Please see http://www.jetbrains.net/confluence/display/TCD4/TeamCity+NUnit+Test+Launcher for details.

If the error is still reproducing, please attach the way you start tests runner and the error message/stacktrace you get from it.

0
Comment actions Permalink

Our solution is written in .NET 3.5. I executed the tests with the above msbuild script and TeamCity configuration. In TC is .NET 3.5 selected and in the msbuild script is for the .net version switch ANY passed (<Exec Command="$(TCNunitLauncher) ANY x86 NUnit-2.4.8 /category-include:$(TestCategory) $(NUnitAssemblies)" />).

This is what was logged to TC's build log:

[15:04:39]: Starting build process in D:\TeamCity\buildAgent\work\80ef1bfad86f8c07
[15:04:41]: Project "productALM4Temp.msbuild.teamcity.patch.tcprojx" (UnitTests target(s)): (1m:14s)
[15:04:42]: [Project "productALM4Temp.msbuild.teamcity.patch.tcprojx" (UnitTests target(s)):] C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe ANY x86 NUnit-2.4.8 /category-include:teamcity bin\Debug\company.product.core.NUnit.dll
[15:04:45]: Appling ordering for tests.
[15:04:50]: company.product.core.NUnit.dll (1m:04s)
[15:04:50]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox.TestFour (8s)
[15:04:58]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox.TestOne (8s)
[15:05:06]: [company.product.core.NUnit.SandBox.TestOne] Test(s) failed. Expected: True But was: False
[15:05:06]: [company.product.core.NUnit.SandBox.TestOne] Test(s) failed. Expected: True
But was: False

at NUnit.Framework.Assert.That(Object actual, Constraint constraint, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition)
at company.product.core.NUnit.SandBox.TestOne() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox.cs:line 16
[15:05:06]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox.TestThree (7s)
[15:05:14]: [company.product.core.NUnit.SandBox.TestThree] Test(s) failed. Expected: True But was: False
[15:05:14]: [company.product.core.NUnit.SandBox.TestThree] Test(s) failed. Expected: True
But was: False

at NUnit.Framework.Assert.That(Object actual, Constraint constraint, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition)
at company.product.core.NUnit.SandBox.TestThree() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox.cs:line 30
[15:05:14]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox.TestTwo (7s)
[15:05:22]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox2.TestFour (7s)
[15:05:30]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox2.TestOne (7s)
[15:05:38]: [company.product.core.NUnit.SandBox2.TestOne] Test(s) failed. Expected: True But was: False
[15:05:38]: [company.product.core.NUnit.SandBox2.TestOne] Test(s) failed. Expected: True
But was: False

at NUnit.Framework.Assert.That(Object actual, Constraint constraint, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition)
at company.product.core.NUnit.SandBox2.TestOne() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox2.cs:line 14
[15:05:38]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox2.TestThree (7s)
[15:05:46]: [company.product.core.NUnit.SandBox2.TestThree] Test(s) failed. Expected: True But was: False
[15:05:46]: [company.product.core.NUnit.SandBox2.TestThree] Test(s) failed. Expected: True
But was: False

at NUnit.Framework.Assert.That(Object actual, Constraint constraint, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition, String message, Object[] args)
at NUnit.Framework.Assert.IsTrue(Boolean condition)
at company.product.core.NUnit.SandBox2.TestThree() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox2.cs:line 28
[15:05:46]: [company.product.core.NUnit.dll] company.product.core.NUnit.SandBox2.TestTwo (7s)
[15:05:54]: Run all unordered tests.
[15:05:55]: company.product.core.NUnit.dll
[15:05:56]: Tests run: 8, Failures: 4, Not run: 0, Time: 0.000 seconds

Test Case Failures:
1) company.product.core.NUnit.SandBox.TestOne : Expected: True
But was: False

at company.product.core.NUnit.SandBox.TestOne() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox.cs:line 16

2) company.product.core.NUnit.SandBox.TestThree : Expected: True
But was: False

at company.product.core.NUnit.SandBox.TestThree() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox.cs:line 30

3) company.product.core.NUnit.SandBox2.TestOne : Expected: True
But was: False

at company.product.core.NUnit.SandBox2.TestOne() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox2.cs:line 14

4) company.product.core.NUnit.SandBox2.TestThree : Expected: True
But was: False

at company.product.core.NUnit.SandBox2.TestThree() in d:\TeamCity\buildAgent\work\d9f2a5bf9941858b\core.NUnit\SandBox2.cs:line 28


[15:05:56]: Some tests has failed in D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\bin\Debug\company.product.core.NUnit.dll, tests run continued. Child test failed
[15:05:56]: 1 Test(s) failed.
[15:05:56]: [Project "productALM4Temp.msbuild.teamcity.patch.tcprojx" (UnitTests target(s)):] D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild(18, 5): error MSB3073: The command "C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe ANY x86 NUnit-2.4.8 /category-include:teamcity bin\Debug\company.product.core.NUnit.dll" exited with code 1.
[15:05:56]: MSBuild output: c:\BuildAgent\bin\..\plugins\dotnetPlugin\bin\JetBrains.BuildServer.MsBuildBootstrap.exe @xml D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild.teamcity.bootstrap.properties TeamCity MSBuild bootstrap v4.0 Copyright (C) JetBrains s.r.o. Start MSBuild... 'c:\WINDOWS\Microsoft.NET\Framework\v3.5\MSBuild.exe' '@"D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild.teamcity.msbuild.tcargs"' working dir = 'D:\TeamCity\buildAgent\work\80ef1bfad86f8c07' Microsoft (R) Build Engine Version 3.5.30729.1 [Microsoft .NET Framework, Version 2.0.50727.3082] Copyright (C) Microsoft Corporation 2007. All rights reserved. Build started 30.04.2009 15:04:41. Project "D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild.teamcity.patch.tcprojx" on node 0 (UnitTests target(s)). D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild(18,5): error MSB3073: The command "C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe ANY x86 NUnit-2.4.8 /category-include:teamcity bin\Debug\company.product.core.NUnit.dll" exited with code 1. Done Building Project "D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild.teamcity.patch.tcprojx" (UnitTests target(s)) -- FAILED. Build FAILED. "D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild.teamcity.patch.tcprojx" (UnitTests target) (1) -> (UnitTests target) -> D:\TeamCity\buildAgent\work\80ef1bfad86f8c07\productALM4Temp.msbuild(18,5): error MSB3073: The command "C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe ANY x86 NUnit-2.4.8 /category-include:teamcity bin\Debug\company.product.core.NUnit.dll" exited with code 1. 0 Warning(s) 1 Error(s) Time Elapsed 00:01:14.60
[15:05:56]: Process exit code: 1
[15:05:57]: Build finished

It says 'Appling ordering for tests' but the first executed test is always a successful one.

0
Comment actions Permalink

Test reordering works only on classes.
Have you used personal builds?

0
Comment actions Permalink

No, have not used personal builds. Everything is committed and the builds are triggered via the web ui.
I have two classes (now).
Class one has four successful tests (Assert.IsTrue(true))
Class two has two successful tests (Assert.IsTrue(true)) and two failing tests (Assert.IsTrue(false)).

Class one is always the first executed test (i've triggered the build three times in a row).

Well, i'm still helpless.

0
Comment actions Permalink

Could you please set system property 'system.teamcity.agent.dotnet.debug' for your build configuraion and re-run that build. Please attach produced build log.

0
Comment actions Permalink

Here we go...



Attachment(s):
Build.log
0
Comment actions Permalink

Thanks!
Could you please add contents of the file from msbuild property $(teamcity_tests_recentlyFailedTests_file)
TeamCity creates that file for build runner to apply tests ordering.
NUnit tests ordering does not apply any order for tests listed in that file. That could be a case
where all test classes are listed in that generated file.

On the build agent, please replace
  <agent>/plugins/dotnetplugin/bin/teamcity-log4net.xml with <agent>/plugins/dotnetplugin/bin/teamcity-log4net-debug.xml

Run build again.

Note, this file is generated for every new build.

0
Comment actions Permalink

The issue was in the name checkioning. It checks the name of test using startsWith, thus snadbox and sandbox2 are seems to be the same.
I'll fix that isse.
Please let me know if you need a patch!

0
Comment actions Permalink

Right! I renamed class Sandbox2 to another Name (SaendeliKasten) and see, it works! Classes get reordered. (still everything in debug mode). How would a patch look like? Just replacing a file somewhere?

0

Please sign in to leave a comment.