Run recently failed tests first option does not appear to be reordering tests correctly
Answered
I've seen a little information on this feature here and there... I have a large test suite using NUnit. I have it integrated into the builds we run as a team on TeamCity.
Our build step currently has these settings:
Runner type: NUnit
NUnit Runner: NUnit 2.6.4
.NET Runtime: x86, V4.0
Have the Reduce test failure feedback time checkbox selected.
We tried this out on a few builds on some branches here and there. We have an auto retry trigger so the failed builds run back to back. After seemingly making no different with the option on, I decided to make a test project with some smaller tests to see how it would act, and it still doesn't seem to run the tests in any different order. Here's the log:
[09:53:06][Step 9/12] Starting: C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe #TeamCityImplicit
[09:53:06][Step 9/12] in directory: C:\BuildAgent\work\3d72ab7f2825ed40
[09:53:07][Step 9/12] Start TeamCity NUnit Test Runner
[09:53:07][Step 9/12] Running NUnit-2.6.4 tests under .NET Framework v4.0 x86
[09:53:07][Step 9/12] TeamCity Test Reordering. Phase 1: Running selected tests.
[09:53:07][Step 9/12] Automation.Tests.dll (15s)
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback.SandBox1
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback.SandBox2
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback.SandBox2] Test(s) failed.
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback.SandBox2] Test(s) failed. at NUnit.Framework.Assert.Fail(String message, Object[] args) at Automation.Tests.Components.Home.ReduceFeedback.SandBox2() in C:\BuildAgent\work\3d72ab7f2825ed40\test\Automation\Automation.Tests\Components\Home\ReduceFeedback.cs:line 21
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback.SandBox3
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback.SandBox4
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback.SandBox4] Test(s) failed.
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback.SandBox4] Test(s) failed. at NUnit.Framework.Assert.Fail(String message, Object[] args) at Automation.Tests.Components.Home.ReduceFeedback.SandBox4() in C:\BuildAgent\work\3d72ab7f2825ed40\test\Automation\Automation.Tests\Components\Home\ReduceFeedback.cs:line 33
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback2.SandBox1
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback2.SandBox1] Test(s) failed.
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback2.SandBox1] Test(s) failed. at NUnit.Framework.Assert.Fail(String message, Object[] args) at Automation.Tests.Components.Home.ReduceFeedback2.SandBox1() in C:\BuildAgent\work\3d72ab7f2825ed40\test\Automation\Automation.Tests\Components\Home\ReduceFeedback.cs:line 42
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback2.SandBox2
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback2.SandBox3
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback2.SandBox3] Test(s) failed.
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback2.SandBox3] Test(s) failed. at NUnit.Framework.Assert.Fail(String message, Object[] args) at Automation.Tests.Components.Home.ReduceFeedback2.SandBox3() in C:\BuildAgent\work\3d72ab7f2825ed40\test\Automation\Automation.Tests\Components\Home\ReduceFeedback.cs:line 54
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback2.SandBox4
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback3.SandBox1
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback3.SandBox2
[09:53:22][Automation.Tests.dll] Automation.Tests.Components.Home.ReduceFeedback3.SandBox3
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback3.SandBox3] Test(s) failed.
[09:53:22][Automation.Tests.Components.Home.ReduceFeedback3.SandBox3] Test(s) failed. at NUnit.Framework.Assert.Fail(String message, Object[] args) at Automation.Tests.Components.Home.ReduceFeedback3.SandBox3() in C:\BuildAgent\work\3d72ab7f2825ed40\test\Automation\Automation.Tests\Components\Home\ReduceFeedback.cs:line 81
[09:53:22][Step 9/12] TeamCity Test Reordering. Phase 2: Running the rest of the tests.
SandBox1, despite passing in the previous build, is still run before SandBox2, which failed in the previous build.
After doing some research, I found that NUnitLauncher (which is being called int he first line of the build log) may be the culprit, but I am not sure how to change from NUnit launcher to something else. Am I going to need to run a command line step and construct my own command to avoid running the launcher?
Please sign in to leave a comment.
Hello Tyler,
What TeamCity version is used? Do you run two subsequent builds on the same branch? We have the bug reported https://youtrack.jetbrains.com/issue/TW-43910, is it your case?
For NUnit versions prior to 3.x TeamCity uses its own NUnit tests launcher. If you want to use your own NUnit console runner, then you should switch to NUnit 3.x. For more details please see our Getting Started with NUnit 3.
Hello Alina,
We are using TeamCity 9.1.6 (build 37459)
We currently do run two subsequent builds of the same branch if the first build fails. We were wanting a method to run the failed tests from the previous build first in the next build, and it just didn't seem like it was working at all. However, we are using feature branches, as TW-43910 mentions, so I think that is actually the same issue.