Known Failure Mecahnism for Unit Tests

Typically in a development cycle unit tests become broken and will stay broken for some time. This may have been a conscious effort to do so and we don't want these tests to show up as failures in the unit test report. Plus, we don't want to comment these out - we still want these recorded somewhere.
We handle this now by throwing a particular exception and our customized test runner removes these from our report. However we ant to use TeamWare to automate the whole process so we don't have a customized test runner. Therefore it would be nice if TeamWare had a similar mechanism and the unit test report could have a section in it for "known to fail" unit tests.
I wish JUnit would have a "known failure" exception that we could throw but I haven't seen one. I guess TeamWare could create some sort of exception we could throw or there may be some other mechanism. Have any ideas?

1 comment
Comment actions Permalink

It is a bit of a dramatic change, but TestNG can handle this scenario with groups. You can tag tests know to fail with the group "fails" and then exclude them from the run or run them with the directive not to fail the build upon test failure.

So for example, if you have "smoketests" that get run on every checkin and "failstests" to indicate that the test is not expected to run, you might have:

test1 - smoketests
test2 - smoketests, failstests
test3 - smoketests

And then have a build triggered by checkin:

groupsinclude="smoketests" groupsexclude="failstests"

if you want to still run the tests, but not fail, you can add the additional run of:

groupsinclude="failstests" haltonfailure="false"

--Tim

0

Please sign in to leave a comment.