NUnit tests fail, but additional build steps still run

Afternoon,

     We have our TeamCity enviroment set up to commit it's artifacts to SVN after a build has been done and tested.

     We used to do this though MSBuild scripts, but have switched to using additional build steps.  Once thing we have noticed, is that if any NUnit tests fail, the next build steps still run.  This means we commit dll's which we know are broken, as tests have failed, not what we want to do.  When using MSBuild, if a target errored, the script stopped.

     Is there a way to get this to happen in team city?  Some way for TeamCity not to do the next steps if the tests fail?

     I suspect we could add a new build configuration which only runs if the main build runs, but this means checking out another copy of the project, and an additional build after each commit.

     Any ideas?

Craig

15 comments
Comment actions Permalink

Hi Craig

By some reason TeamCity doesn't catch rest results while it should.
Could you please show how build step with NUnit is configured, and a resulted build log.

Michael

0
Comment actions Permalink

Team city knows the tests have failed, and says so.

Abridged output from where the tests failed


[09:49:41]: [Step 2/5] Some tests has failed in C:\BuildAgent\work\ce1dc1548b85904c\dev\Tests\bin\Release\MKM.PDL.VLE.Tests.dll, tests run continued. Child test failed
[09:49:42]: [Step 2/5] ***************** End Program Output *****************
[09:49:42]: [Step 2/5] ##teamcity[importData type='dotNetCoverage' tool='ncover' file='C:\BuildAgent\temp\buildTmp\tmp99A3.tmp']
[09:49:42]: [Step 2/5] Importing data from 'C:\BuildAgent\temp\buildTmp\tmp99A3.tmp' with 'dotNetCoverage' processor
[09:49:42]: [Step 2/5] Process exited with code 0
[09:49:42]: Step 3/5
[09:49:42]: [Step 3/5] Using runner: simpleRunner
[09:49:42]: [Step 3/5] Starting: c:\svn.exe update C:\projects\MKMdllsRepos\PDL\VLE\ --username * --password * --non-interactive
[09:49:42]: [Step 3/5] in directory: C:\BuildAgent\work\ce1dc1548b85904c
[09:49:43]: [Step 3/5] At revision 1201.
[09:49:43]: [Step 3/5] Process exited with code 0


NUnit Setup:

NUnit runner:                         NUnit 2.5.8                         NUnit 2.5.7                         NUnit 2.5.6                         NUnit 2.5.5                         NUnit 2.5.4                         NUnit 2.5.3                         NUnit 2.5.2                         NUnit 2.5.0                         NUnit 2.4.8                         NUnit 2.4.7                         NUnit 2.4.6                         NUnit 2.4.1                         NUnit 2.2.10                 
.NET Runtime: Platform:                         auto (MSIL)                         x86                         x64                 
Version:                         v2.0                         v4.0                 
Run tests from:
Edit assemblies include list Edit assemblies include list:
dev\Tests\bin\Release\MKM.PDL.VLE.Tests.dll
Hide
Enter comma or new line separated paths to assembly files relative to checkout directory. Wildcards are supported.



Thats all that's set.

0
Comment actions Permalink

Hi Craig

Is Fail build if at least one test failed option enabled on General settings page?

If so, could you please send us full build log - here or to teamcity-feedback@jetbrains.com (with a link to this thread).

Thanks
Michael

0
Comment actions Permalink

Yep, that's selected.  I'll email the details.  Thanks.

0
Comment actions Permalink

I have the same problem with unit test failing but subsequent build steps continuing. I have noticed that for custom tasks, batches or executables setting the build status to failure or error is insufficient, and I have to ensure that the process returns a non-zero exit code to cause the build to terminate at that step.

Effectively, we need the option to be able to terminate the entire build process if a give step fails. My process is relatively complex:

  1. Update assembly versions with the build format
  2. Compile.
  3. Run unit tests and generate coverage reports.
  4. Run duplicate code finder.
  5. Sanitize any configuration files and extract static config from configuration server.
  6. Build installer msi/exe.
  7. Optionall (custom build parameter) tag and publish build to deployment server.


Now at this point, if the unit tests fail or the configuration of the unit test assembly is faulty (not correctly set), the build should exit without continuing to subsequent steps. Ideally, each build step should have a flag to fail the build configuration, but at the very least the general settings to fail build below should apply (all of which I have selected).

  • build process exit code is not zero
  • at least one test failed
  • an error message is logged by build runner


Note that all our build configurations are based on a template, so these settings are inherited - not sure whether or not that will aid you in debugging this.

0
Comment actions Permalink

I believe the culprit in this case was the code coverage.  It looks like the tests returned a non zero exit code, but the code coverage returned a 0 code. We restructured all our builds to not do code coverage on this build, and that seemed to work.

0
Comment actions Permalink

I need code coverage on all builds, and I'm not going to create two sets of build steps (one for tagging and one that executes on each check in), and then have code coverage on only some of them.

0
Comment actions Permalink

As a test, I removed the dotCover configuration and found that subsequent steps still ran even though the tests failed. dotCover is a red herring - the failure seems to be either in the integration with the build runner for nUnit (or in nUnit itself).

0
Comment actions Permalink

Hi Craig

Sorry, I missed you reply.
The problem is caused by our NUnit runner that doesn't build step failure. It happens even if coverage is disabled.
Adam already posted this issue in TW-17002, you can watch for it.

Thanks

0
Comment actions Permalink

We have the same problem as described. We are not using code coverage tools and have the "fail if at least one test fails" option set. I thought it might be either the combination of options and team city is reporting incorrectly, or possibly, the version of NUnit - we use the latest release from the web site though.

Help/advice would be appreciated, I can provide the build log if required.

Paul Shannon

0
Comment actions Permalink

Hi Paul

How are you rinnong Nunit? could you post screenshots please.

Also, we made a few related fixes in version 6.5.2.

Michael

0
Comment actions Permalink

Hi Michael,

I'm a colleague of Paul Shannon.

We've upgraded to 6.5.2, but are still experiencing this issue.

I've attached screenshot of the log failing, but continuing, our NUnit Settings and our General settings.

Let me know if there is anything else we can supply.

This is currently quite severe and is preventing us from moving to Team City.

Ronnie



Attachment(s):
General Settings.png
NUnit Settings.png
Continuing on Failed Test.png
0
Comment actions Permalink

Also, the thread dump for the NUnit Launcher:


Thread dumps of Api :: Build Pipeline #16 (14 Jul 11 12:22)

Process ID (PID): 11324

Process command line

Process is running under CLR version: v2.0.50727

Thread id=3704
at NUnit.Core.TestRunnerThread.Wait() 
at NUnit.Core.ThreadedTestRunner.Wait() 
at NUnit.Core.ThreadedTestRunner.EndRun() 
at NUnit.Core.ThreadedTestRunner.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.ProxyTestRunner.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.RemoteTestRunner.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(System.RuntimeMethodHandle md, System.Object[] args, System.Object server, System.Int32 methodPtr, System.Boolean fExecuteInContext, System.Object[]& outArgs) 
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage msg, System.Int32 methodPtr, System.Boolean fExecuteInContext) 
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage msg) 
at System.Runtime.Remoting.Messaging.ServerObjectTerminatorSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage reqMsg) 
at System.Runtime.Remoting.Messaging.ServerContextTerminatorSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage reqMsg) 
at System.Runtime.Remoting.Channels.CrossContextChannel.SyncProcessMessageCallback(System.Object[] args) 
at System.Threading.Thread.CompleteCrossContextCallback(System.Threading.InternalCrossContextDelegate ftnToCall, System.Object[] args) 
at System.Threading.Thread.InternalCrossContextCallback(System.Runtime.Remoting.Contexts.Context ctx, System.Threading.InternalCrossContextDelegate ftnToCall, System.Object[] args) 
at System.Runtime.Remoting.Channels.CrossContextChannel.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage reqMsg) 
at System.Runtime.Remoting.Channels.ChannelServices.SyncDispatchMessage(System.Runtime.Remoting.Messaging.IMessage msg) 
at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoDispatch(System.Byte[] reqStmBuff, System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage smuggledMcm, System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage& smuggledMrm) 
at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatchCallback(System.Object[] args) 
at System.Threading.Thread.CompleteCrossContextCallback(System.Threading.InternalCrossContextDelegate ftnToCall, System.Object[] args) 
AD Switch
at System.Runtime.Remoting.Channels.CrossAppDomainSink.DoTransitionDispatch(System.Byte[] reqStmBuff, System.Runtime.Remoting.Messaging.SmuggledMethodCallMessage smuggledMcm, System.Runtime.Remoting.Messaging.SmuggledMethodReturnMessage& smuggledMrm) 
at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage reqMsg) 
at System.Runtime.Remoting.Proxies.RemotingProxy.CallProcessMessage(System.Runtime.Remoting.Messaging.IMessageSink ms, System.Runtime.Remoting.Messaging.IMessage reqMsg, System.Runtime.Remoting.Contexts.ArrayWithSize proxySinks, System.Threading.Thread currentThread, System.Runtime.Remoting.Contexts.Context currentContext, System.Boolean bSkippingContextChain) 
at System.Runtime.Remoting.Proxies.RemotingProxy.InternalInvoke(System.Runtime.Remoting.Messaging.IMethodCallMessage reqMcmMsg, System.Boolean useDispatchMessage, System.Int32 callType) 
at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage reqMsg) 
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData& msgData, System.Int32 type) 
at JetBrains.TeamCity.NUnitLauncher.NUnitRunner.Run(NUnit.Core.TestRunner runner, NUnit.Core.EventListener logger, JetBrains.TeamCity.NUnitCommon.NUnit.NUnitTestArguments args, JetBrains.TeamCity.NUnitLauncher.ITestOrderingStrategy info) in NUnitRunner.cs:line 62
at JetBrains.TeamCity.NUnitLauncher.AssemblyTest.Run(JetBrains.TeamCity.NUnitLauncher.NUnitEventListener listener, JetBrains.TeamCity.NUnitLauncher.ITestOrderingStrategy strategy) in AssemblyTest.cs:line 90
at JetBrains.TeamCity.NUnitLauncher.RunTestAction.Action(JetBrains.TeamCity.NUnitLauncher.AssemblyTest test) in RunTestAction.cs:line 23
at JetBrains.TeamCity.NUnitLauncher.TryOneAssemblyTest.Action() in TryOneAssemblyTest.cs:line 19
at JetBrains.TeamCity.NUnitLauncher.TryOneAssembly.Do() in TryOneAssembly.cs:line 29
at JetBrains.TeamCity.NUnitLauncher.NUnitRunnerBase.RunInternal() in NUnitRunnerBase.cs:line 138
at JetBrains.TeamCity.NUnitLauncher.NUnitRunnerBase.Run() in NUnitRunnerBase.cs:line 60
at JetBrains.TeamCity.NUnitLauncher.NUnitTestRunnerImpl.Run() in NUnitTestRunnerImpl.cs:line 84
at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.ExecuteTasks(JetBrains.TeamCity.NUnitCommon.Arguments.ITestRunArguments arguments, JetBrains.TeamCity.NUnitCommon.IPlugableTestRunner[] plugRuns) in NUnitLauncher2.cs:line 133
at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.Run2(System.String[] args) in NUnitLauncher2.cs:line 112
at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.Run(System.String[] args) in NUnitLauncher2.cs:line 54
at JetBrains.Container.ComponentsExecutable.Main(System.String[] args, JetBrains.Container.ILoggerInitializer loggerInitializer, System.Reflection.Assembly[] assemblies) in ComponentsExecutable.cs:line 24
at JetBrains.Container.ComponentsExecutable.Main(System.String[] args, System.Reflection.Assembly[] assemblies) in ComponentsExecutable.cs:line 10
at JetBrains.BuildServer.NAntLoggers.NUnitLauncher2.Main(System.String[] args) in NUnitLauncher2.cs:line 47


Thread id=12388


Thread id=4296
at System.Threading.WaitHandle.WaitOne(System.Int64 timeout, System.Boolean exitContext) 
at System.Threading.WaitHandle.WaitOne(System.Int32 millisecondsTimeout, System.Boolean exitContext) 
at JetBrains.BuildServer.CommonLoggers.Impl.LoggerThreadBase.Run() in LoggerThreadBase.cs:line 88
at System.Threading.ThreadHelper.ThreadStart_Context(System.Object state) 
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) 
at System.Threading.ThreadHelper.ThreadStart() 


Thread id=6652
at System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[] waitHandles, System.Int32 millisecondsTimeout, System.Boolean exitContext) 
at System.Net.TimerThread.ThreadProc() 
at System.Threading.ThreadHelper.ThreadStart_Context(System.Object state) 
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) 
at System.Threading.ThreadHelper.ThreadStart() 


Thread id=4112


Thread id=12004
at System.Threading.Monitor.Wait(System.Object obj, System.Int32 millisecondsTimeout, System.Boolean exitContext) 
at NUnit.Core.EventPump.PumpThreadProc() 
at System.Threading.ThreadHelper.ThreadStart_Context(System.Object state) 
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) 
at System.Threading.ThreadHelper.ThreadStart() 
AD Switch


Thread id=7348
at Castle.DynamicProxy.Generators.GeneratorUtil.CopyOutAndRefParameters(Castle.DynamicProxy.Generators.Emitters.SimpleAST.TypeReference[] dereferencedArguments, Castle.DynamicProxy.Generators.Emitters.SimpleAST.LocalReference invocation, System.Reflection.MethodInfo method, Castle.DynamicProxy.Generators.Emitters.MethodEmitter emitter) 
at Castle.DynamicProxy.Generators.MethodWithInvocationGenerator.BuildProxiedMethodBody(Castle.DynamicProxy.Generators.Emitters.MethodEmitter emitter, Castle.DynamicProxy.Generators.Emitters.ClassEmitter class, Castle.DynamicProxy.ProxyGenerationOptions options, Castle.DynamicProxy.Generators.INamingScope namingScope) 
at Castle.DynamicProxy.Generators.MethodGenerator.Generate(Castle.DynamicProxy.Generators.Emitters.ClassEmitter class, Castle.DynamicProxy.ProxyGenerationOptions options, Castle.DynamicProxy.Generators.INamingScope namingScope) 
at Castle.DynamicProxy.Contributors.CompositeTypeContributor.ImplementMethod(Castle.DynamicProxy.Generators.MetaMethod method, Castle.DynamicProxy.Generators.Emitters.ClassEmitter class, Castle.DynamicProxy.ProxyGenerationOptions options, Castle.DynamicProxy.Contributors.OverrideMethodDelegate overrideMethod) 
at Castle.DynamicProxy.Contributors.CompositeTypeContributor.Generate(Castle.DynamicProxy.Generators.Emitters.ClassEmitter class, Castle.DynamicProxy.ProxyGenerationOptions options) 
at Castle.DynamicProxy.Generators.InterfaceProxyWithoutTargetGenerator.GenerateType(System.String typeName, System.Type proxyTargetType, System.Type[] interfaces, Castle.DynamicProxy.Generators.INamingScope namingScope) 
at Castle.DynamicProxy.Generators.InterfaceProxyWithTargetGenerator.GenerateCode(System.Type proxyTargetType, System.Type[] interfaces, Castle.DynamicProxy.ProxyGenerationOptions options) 
at Castle.DynamicProxy.DefaultProxyBuilder.CreateInterfaceProxyTypeWithoutTarget(System.Type interfaceToProxy, System.Type[] additionalInterfacesToProxy, Castle.DynamicProxy.ProxyGenerationOptions options) 
at Castle.DynamicProxy.ProxyGenerator.CreateInterfaceProxyWithoutTarget(System.Type interfaceToProxy, System.Type[] additionalInterfacesToProxy, Castle.DynamicProxy.ProxyGenerationOptions options, Castle.DynamicProxy.IInterceptor[] interceptors) 
at Moq.Proxy.CastleProxyFactory.CreateProxy<Api.WebApp.Controllers.IFinanceServiceFactory>(Moq.Proxy.ICallInterceptor interceptor, System.Type[] interfaces, System.Object[] arguments) 
at Moq.Mock<System.__Canon>.<InitializeInstance>b__0() 
at Moq.PexProtector.Invoke(System.Action action) 
at Moq.Mock<System.__Canon>.InitializeInstance() 
at Moq.Mock<System.__Canon>.OnGetObject() 
at Moq.Mock<Api.WebApp.Controllers.IFinanceServiceFactory>.get_Object() 
at Api.WebApp.Tests.Controllers.GivenAFinanceCalculationWithAnInvalidContentType.ThenTheErrorPageIsReturned() in FinanceControllerTests.cs:line 103
at System.RuntimeMethodHandle.InvokeMethodFast(System.Object target, System.Object[] arguments, System.Signature sig, System.Reflection.MethodAttributes methodAttributes, System.RuntimeTypeHandle typeOwner) 
at System.Reflection.RuntimeMethodInfo.Invoke(System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture, System.Boolean skipVisibilityChecks) 
at System.Reflection.RuntimeMethodInfo.Invoke(System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) 
at NUnit.Core.Reflect.InvokeMethod(System.Reflection.MethodInfo method, System.Object fixture, System.Object[] args) 
at NUnit.Core.TestMethod.RunTestMethod(NUnit.Core.TestResult testResult) 
at NUnit.Core.TestMethod.doTestCase(NUnit.Core.TestResult testResult) 
at NUnit.Core.TestMethod.doRun(NUnit.Core.TestResult testResult) 
at NUnit.Core.TestMethod.Run(NUnit.Core.TestResult testResult) 
at NUnit.Core.NUnitTestMethod.Run(NUnit.Core.TestResult testResult) 
at NUnit.Core.TestMethod.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.RunAllTests(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestFixture.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.RunAllTests(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.RunAllTests(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.RunAllTests(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.RunAllTests(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.RunAllTests(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.RunAllTests(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.TestResult suiteResult, NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestSuite.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.SimpleTestRunner.Run(NUnit.Core.EventListener listener, NUnit.Core.ITestFilter filter) 
at NUnit.Core.TestRunnerThread.TestRunnerThreadProc() 
at System.Threading.ThreadHelper.ThreadStart_Context(System.Object state) 
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) 
at System.Threading.ThreadHelper.ThreadStart() 
AD Switch






0
Comment actions Permalink

Hi,
this post is rather old and I gather this should be fixed but I'll respond here since we are experiencing the same issue.
Build Step 2 is to run Unit Tests + Test coverage. Even though test(s) fail, the build will continue to Build step 3.

One can notice that the code coverage process exits with 0, maybe it does have something to do with that.

We are running on 7.0 (build 21241).

Attached is an overview of the build steps
build steps.PNG

Kind regards
Frank

0
Comment actions Permalink

We have same bug in latest version (2017.2). Opened a ticket  https://youtrack.jetbrains.com/issue/TW-54300

0

Please sign in to leave a comment.