Build Script Interaction -- testFailed reporting

We are using a Nant script to run full-product end-to-end tests on our software.  A while ago we had the test reporting working but now it seems as though all tests are reported as successful even if they fail.  I tried adding flowIDs as suggested by results I found while searching, but with no success.  I tried with the same flowId and different ones, as I'll post below.  The example I ran was just a 2-test suite, the first of which fails and the 2nd passes.  But in both cases, TeamCity reports both tests as okay.

---------------------------------------------------------------------------- This test is using flowId's associated with each test ---------------------------------------------------------

[12:21:00][NAnt output] ##teamcity[testStarted name='OpenAndClose1' flowId='OpenAndClose1' captureStandardOutput='true']

[12:21:00][NAnt output] [] Done.

[12:21:00][NAnt output] [] 1c1

[12:21:00][NAnt output] [] < hia

[12:21:00][NAnt output] [] \ No newline at end of file

[12:21:00][NAnt output] [] ---

[12:21:00][NAnt output] [] > hi

[12:21:00][NAnt output] []

[12:21:00][NAnt output] []

[12:21:00][NAnt output] []

[12:21:00][NAnt output] ##teamcity[testFailed name='OpenAndClose1' flowId='OpenAndClose1' details='test results differ']

[12:21:00][NAnt output] [ControlConsoleTest]

[12:21:00][NAnt output] ##teamcity[testFinished name='OpenAndClose1' flowId='OpenAndClose1']

[12:21:00][NAnt output] [ControlConsoleTest]

[12:21:00][NAnt output] [ControlConsoleTest] Test Failed:  OpenAndClose1

[12:21:00][NAnt output] [ControlConsoleTest]

[12:21:00][NAnt output] ##teamcity[testStarted name='OpenAndClose2' flowId='OpenAndClose2' captureStandardOutput='true']

[12:21:00][NAnt output] [] Done.

[12:21:00][NAnt output] [] Files C:\BuildAgent\work\aae1fb4bc9febbb2\Testing\...\out.log and C:\BuildAgent\work\aae1fb4bc9febbb2\temp\out.log are identical

[12:21:00][NAnt output] []

[12:21:00][NAnt output] []

[12:21:00][NAnt output] [] Files are identical. Test passed

[12:21:00][NAnt output] [ControlConsoleTest]

[12:21:00][NAnt output] ##teamcity[testFinished name='OpenAndClose2' flowId='OpenAndClose2']

[12:21:00][NAnt output] [ControlConsoleTest]

[12:21:00][NAnt output]

[12:21:00][NAnt output] BUILD FAILED - 4 non-fatal error(s), 0 warning(s)

[12:21:00][NAnt output]

[12:21:00][NAnt output] C:\BuildAgent\work\aae1fb4bc9febbb2\ControlConsole.build(181,10):

[12:21:00][NAnt output] Not all tests passed.




---------------------------------------------------------------------------- This test is using one flowId ---------------------------------------------------------

[12:43:34][NAnt output] ##teamcity[testStarted name='OpenAndClose1' flowId='1' captureStandardOutput='true']

[12:43:34][NAnt output] [] Done.

[12:43:34][NAnt output] [] 1c1

[12:43:34][NAnt output] [] < hia

[12:43:34][NAnt output] [] \ No newline at end of file

[12:43:34][NAnt output] [] ---

[12:43:34][NAnt output] [] > hi

[12:43:34][NAnt output] []

[12:43:34][NAnt output] []

[12:43:34][NAnt output] []

[12:43:34][NAnt output] ##teamcity[testFailed name='OpenAndClose1' flowId='1' details='test results differ']

[12:43:34][NAnt output] [ControlConsoleTest]

[12:43:34][NAnt output] ##teamcity[testFinished name='OpenAndClose1' flowId='1']

[12:43:34][NAnt output] [ControlConsoleTest]

[12:43:34][NAnt output] [ControlConsoleTest] Test Failed:  OpenAndClose1

[12:43:34][NAnt output] [ControlConsoleTest]

[12:43:34][NAnt output] ##teamcity[testStarted name='OpenAndClose2' flowId='1' captureStandardOutput='true']

[12:43:34][NAnt output] [] Done.

[12:43:34][NAnt output] [] Files C:\BuildAgent\work\aae1fb4bc9febbb2\Testing\...\Golden\out.log and C:\BuildAgent\work\aae1fb4bc9febbb2\temp\out.log are identical

[12:43:34][NAnt output] []

[12:43:34][NAnt output] []

[12:43:34][NAnt output] [] Files are identical. Test passed

[12:43:34][NAnt output] [ControlConsoleTest]

[12:43:34][NAnt output] ##teamcity[testFinished name='OpenAndClose2' flowId='1']

[12:43:34][NAnt output] [ControlConsoleTest]

[12:43:34][NAnt output]

[12:43:34][NAnt output] BUILD FAILED - 4 non-fatal error(s), 0 warning(s)

[12:43:34][NAnt output]

[12:43:34][NAnt output] C:\BuildAgent\work\aae1fb4bc9febbb2\ControlConsole.build(181,10):

[12:43:34][NAnt output] Not all tests passed.

[12:43:34][NAnt output]

[12:43:34][NAnt output] Total time: 35.4 seconds.




In both cases, TeamCity lists both tests as okay:
pic.png

In this case, the goal is to report all the failed tests while still completing the entire build, and then reporting on the success of the build at the end.  I can get the build to run through the failures and fail at the end properly, but we still don't get to see which tests failed unless we dig into the log file.  Does anyone see what I'm doing wrong?  The build interaction messages are being sent as Log(Level.Info) calls from a custom C# nant task.  I assume they are getting there properly since TeamCity lists the tests by name.
6 comments
Comment actions Permalink

Hello,

Could you please specify another flowId attribute value, i.e. flowId='nantWrappingTests'.

I do not see in the logs where you call

##teamcity[testFailed type='comparisonFailure'
 
name='test2'
 
message='failure message'
 
details='message and stack trace'
 
expected='expected value'
 
actual='actual value']


or

##teamcity[testFailed name='test1'
 
message='failure message'
 
details='message and stack trace']


service message to mark test as failed.

For more details, please take a look at the doc page:
http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingTests

0
Comment actions Permalink

Eugene,

Thanks for the feedback.  I can try a different flowId but I don't see why that would matter.  

As I said in my original post, there should _not_ be a failure message for test2 as test2 succeeds.  There is only a failure message for test1.  However, teamcity reports that both tests succeeded.

0
Comment actions Permalink

According to our documentation, every testStarted message must be accompanied by testFinished. So if you want to report failed test you still have to use testFinished too:

##teamcity[testStarted name='test1']

##teamcity[testFailed name='test1' message='failure message' details='message and stack trace']

##teamcity[testFinished name='test1']



Please make sure you did not forget to produce testFinished messages.
0
Comment actions Permalink

Pavel, I appreciate you trying to help, as you can clearly see in my original post, here is my log:

[12:21:00][NAnt output] ##teamcity[testStarted name='OpenAndClose1' flowId='OpenAndClose1' captureStandardOutput='true']

.....

[12:21:00][NAnt output] ##teamcity[testFailed name='OpenAndClose1' flowId='OpenAndClose1' details='test results differ']

[12:21:00][NAnt output] [ControlConsoleTest]

[12:21:00][NAnt output] ##teamcity[testFinished name='OpenAndClose1' flowId='OpenAndClose1']


I have a testStarted, testFailed, and testFinished message, but this test is still reported as succeeding.
0
Comment actions Permalink

Ok, sorry, I overlooked it. Could you please attach the whole output of the process? This way we could reproduce problem locally.
As to flowId you need to use it if messages are produced from different processes. If you have single process which outputs messages then there is no need to use flowId.

0
Comment actions Permalink

I finally managed to get it working.  I had to change a bunch of things, so I'm not sure what actually caused the error.  I changed the test suite name from "e2e_tests" to just "tests", in case the name wasn't being parsed right.  I also made the flowId "mainFlow" instead of one having a number.  Finally, I had half of my reporting being done in a custom C# nant task, I took it all out of there and put it in the actual BUILD file.  If I get some time, I'll try to figure out what actually solved the error, but just letting you know that I did it get working, thanks!

0

Please sign in to leave a comment.