Why does the build report success when tests report failure?

Matt Hall wrote:

Hi all,

I'm testing the beta release of TeamCity, I hope this is the right place to ask questions.

I have our project building and the tests running, and TeamCity seems to understand there's a problem when some tests fail because it will send out a build failed email, however in the interface the build is listed as successfully built.

Is there a setting somewhere I can tell teamcity to mark the build failed if any tests fail?

Builds finishes successfully if build error code = 0. When a test fails, we suppose that the
build will fail, but if error code = 0 it finishes as successful (BUILD SUCCESSFUL for Ant).

If it is not your case, please file a Jira request and attach full build log.

Kind regards,
KIR

--
Kirill Maximov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

Builds finishes successfully if build error code =
0. When a test fails, we suppose that the
uild will fail, but if error code = 0 it finishes as
successful (BUILD SUCCESSFUL for Ant).

If it is not your case, please file a Jira request
and attach full build log.

Sorry, I'm a bit confused by your message. Is there a way I should force the build to fail if some of the tests fail? I'd rather not set a haltonfailure="true" property on the junit task as I'd like it to run through all the tests to get a complete list of tests that are failing.

Thanks very much,

Matt

Matt Hall wrote:
>> Builds finishes successfully if build error code =
>> 0. When a test fails, we suppose that the
>> uild will fail, but if error code = 0 it finishes as
>> successful (BUILD SUCCESSFUL for Ant).
>>
>> If it is not your case, please file a Jira request
>> and attach full build log.

Sorry, I'm a bit confused by your message. Is there a way I should force the build to fail if some of the tests fail? I'd rather not set a haltonfailure="true" property on the junit task as I'd like it to run through all the tests to get a complete list of tests that are failing.

In junit task you can specify a property which will be set if some test fails.
After junit task you can analyze this property and invoke ]]> task.

Hope this helps,
KIR

Thanks very much,

Matt

--
Kirill Maximov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

Matt Hall wrote:
>> Builds finishes successfully if build error code =
>> 0. When a test fails, we suppose that the
>> uild will fail, but if error code = 0 it finishes as
>> successful (BUILD SUCCESSFUL for Ant).
>>
>> If it is not your case, please file a Jira request
>> and attach full build log.

Sorry, I'm a bit confused by your message. Is there a way I should force the build to fail if some of the tests fail? I'd rather not set a haltonfailure="true" property on the junit task as I'd like it to run through all the tests to get a complete list of tests that are failing.

You may also watch/vote for:
http://www.jetbrains.net/jira/browse/TW-670

KIR


--
Kirill Maximov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

I figured out the failureproperty workaround myself, but I have to say that it rather dramatically violates the principle of least surprise. It's probably worth adding a flag to appropriate runner's configuration screens that would report a build failure if any tests fail.

--Dave Griffith

Ignore that. Like an idiot, I hadn't clicked on the issue link.

--Dave Griffith

Thanks Kirill, that conditional fail did the trick. I also did vote for the issue though, as it doesn't make sense to me to receive a build failed email because a test failed, then click the link and see a build succeeded status message on the web interface.

Thanks for the help.

Matt