testFailed message is not recognized as test failure

Hello Philip,

Can you please share the exact code that sends the service messages into stdout? Normally we`d expect this sequence of messages:

##teamcity[testStarted name='MyTest.test1']
##teamcity[testFailed name='MyTest.test1' message='failure message' details='message and stack trace']
##teamcity[testFinished name='MyTest.test1']

Can you please confirm if all three are emitted? 

Hello Fedor,

Thanks for quick response.

I confirm I have used all three service message in correct sequence and also 'testSuiteStarted'

'testSuiteFinished' in begin and at the end.

I tested it in a way to deliberately failed all my test cases and it showed in the screenshot above.

Where  "Test fail!" is 'message' parameter and "here is my test failed details" is the 'details' parameter.

the service message is consumed by teamcity and show no errors as it will complain in stdout if the format is incorrect.

Hello Philip,

I see, thank you - this definitely looks weird. May I ask you to enable XML-RPC debug logging on the agent and try to reproduce the issue, then share the resulting teamcity-xmlrpc.log.* files covering the time of build? You may upload them privately via uploads.jetbrains.com resource - please just let me know the upload ID once done. 

This log contains all communication between agent and server, so hopefully we could see the exact messages sequence. If you could also share the teamcity-server.log for the same period, this would be helpful too (I believe uncaught exceptions  related to the service messages may not be shown in the build log, but they will appear on the server log). 

Hello Fedor,

I uploaded logs in this upload id: 2023_03_15_1gy7FSCXUyVtidZwRqn6uj (file: teamcity_log.zip)

inside there is a `before` and `after` folder that contain logs before and after the build run.

I also made sure that the server is only running one build with one agent at that moment.

I can see some testFailed messages being sent, but not notice any error/exception.

p.s. We also updated TC to  2022.10.2 (build 117025), issue still exists

Thank you

Hello Philip,

I see that for the testFailed message, you have not specified optional parameter flowId whereas two other messages (testStarted and testFinished) do specify this parameter. Due to this, test failure was essentially reported in a different flow and was ignored.

I was able to get the desired behavior by either omitting flowId from all messages, or making sure it is included with each service message for the test:

echo ##teamcity[testStarted name='test' flowId='test chromacity']
echo ##teamcity[testFailed type='testFailure' name='test' message='Test fail!' flowId='test chromacity']
echo ##teamcity[testFinished name='test' duration='123' flowId='test chromacity']

Please let me know if this helps.

Hello Fedor,

Thanks for spotting the problem, I corrected it and it works now.

I wonder if I use it in a nested test case, do I have to use unique flowid for each test case?

Thank you

Hello Philip,

Yes, as our documentation states:

> Starting another test finishes the currently started test in the same flow. To still report tests from within other tests, you will need to specify another flowId in the nested test service messages.

If you run tests sequentially you may safely omit flowId from your messages. In case tests run asynchronously, using flowId is necessary.