NUnit tests fail, but additional build steps still run
Afternoon,
We have our TeamCity enviroment set up to commit it's artifacts to SVN after a build has been done and tested.
We used to do this though MSBuild scripts, but have switched to using additional build steps. Once thing we have noticed, is that if any NUnit tests fail, the next build steps still run. This means we commit dll's which we know are broken, as tests have failed, not what we want to do. When using MSBuild, if a target errored, the script stopped.
Is there a way to get this to happen in team city? Some way for TeamCity not to do the next steps if the tests fail?
I suspect we could add a new build configuration which only runs if the main build runs, but this means checking out another copy of the project, and an additional build after each commit.
Any ideas?
Craig
Please sign in to leave a comment.
Hi Craig
By some reason TeamCity doesn't catch rest results while it should.
Could you please show how build step with NUnit is configured, and a resulted build log.
Michael
Team city knows the tests have failed, and says so.
Abridged output from where the tests failed
[09:49:41]: [Step 2/5] Some tests has failed in C:\BuildAgent\work\ce1dc1548b85904c\dev\Tests\bin\Release\MKM.PDL.VLE.Tests.dll, tests run continued. Child test failed
[09:49:42]: [Step 2/5] ***************** End Program Output *****************
[09:49:42]: [Step 2/5] ##teamcity[importData type='dotNetCoverage' tool='ncover' file='C:\BuildAgent\temp\buildTmp\tmp99A3.tmp']
[09:49:42]: [Step 2/5] Importing data from 'C:\BuildAgent\temp\buildTmp\tmp99A3.tmp' with 'dotNetCoverage' processor
[09:49:42]: [Step 2/5] Process exited with code 0
[09:49:42]: Step 3/5
[09:49:42]: [Step 3/5] Using runner: simpleRunner
[09:49:42]: [Step 3/5] Starting: c:\svn.exe update C:\projects\MKMdllsRepos\PDL\VLE\ --username * --password * --non-interactive
[09:49:42]: [Step 3/5] in directory: C:\BuildAgent\work\ce1dc1548b85904c
[09:49:43]: [Step 3/5] At revision 1201.
[09:49:43]: [Step 3/5] Process exited with code 0
NUnit Setup:
Thats all that's set.
Hi Craig
Is Fail build if at least one test failed option enabled on General settings page?
If so, could you please send us full build log - here or to teamcity-feedback@jetbrains.com (with a link to this thread).
Thanks
Michael
Yep, that's selected. I'll email the details. Thanks.
I have the same problem with unit test failing but subsequent build steps continuing. I have noticed that for custom tasks, batches or executables setting the build status to failure or error is insufficient, and I have to ensure that the process returns a non-zero exit code to cause the build to terminate at that step.
Effectively, we need the option to be able to terminate the entire build process if a give step fails. My process is relatively complex:
Now at this point, if the unit tests fail or the configuration of the unit test assembly is faulty (not correctly set), the build should exit without continuing to subsequent steps. Ideally, each build step should have a flag to fail the build configuration, but at the very least the general settings to fail build below should apply (all of which I have selected).
Note that all our build configurations are based on a template, so these settings are inherited - not sure whether or not that will aid you in debugging this.
I believe the culprit in this case was the code coverage. It looks like the tests returned a non zero exit code, but the code coverage returned a 0 code. We restructured all our builds to not do code coverage on this build, and that seemed to work.
I need code coverage on all builds, and I'm not going to create two sets of build steps (one for tagging and one that executes on each check in), and then have code coverage on only some of them.
As a test, I removed the dotCover configuration and found that subsequent steps still ran even though the tests failed. dotCover is a red herring - the failure seems to be either in the integration with the build runner for nUnit (or in nUnit itself).
Hi Craig
Sorry, I missed you reply.
The problem is caused by our NUnit runner that doesn't build step failure. It happens even if coverage is disabled.
Adam already posted this issue in TW-17002, you can watch for it.
Thanks
We have the same problem as described. We are not using code coverage tools and have the "fail if at least one test fails" option set. I thought it might be either the combination of options and team city is reporting incorrectly, or possibly, the version of NUnit - we use the latest release from the web site though.
Help/advice would be appreciated, I can provide the build log if required.
Paul Shannon
Hi Paul
How are you rinnong Nunit? could you post screenshots please.
Also, we made a few related fixes in version 6.5.2.
Michael
Hi Michael,
I'm a colleague of Paul Shannon.
We've upgraded to 6.5.2, but are still experiencing this issue.
I've attached screenshot of the log failing, but continuing, our NUnit Settings and our General settings.
Let me know if there is anything else we can supply.
This is currently quite severe and is preventing us from moving to Team City.
Ronnie
Attachment(s):
General Settings.png
NUnit Settings.png
Continuing on Failed Test.png
Also, the thread dump for the NUnit Launcher:
Thread dumps of Api :: Build Pipeline #16 (14 Jul 11 12:22)
Process tree:
PID: 11324 C:\TeamCity\buildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe
Process ID (PID): 11324
Process command line
Hi,
this post is rather old and I gather this should be fixed but I'll respond here since we are experiencing the same issue.
Build Step 2 is to run Unit Tests + Test coverage. Even though test(s) fail, the build will continue to Build step 3.
One can notice that the code coverage process exits with 0, maybe it does have something to do with that.
We are running on 7.0 (build 21241).
Attached is an overview of the build steps
Kind regards
Frank
We have same bug in latest version (2017.2). Opened a ticket https://youtrack.jetbrains.com/issue/TW-54300