Using TeamCity service messages to find if a test failed and display on build log

Answered

Hello!

I am running some smoke tests on TeamCity using a batch file. The batch file runs my tests on TestComplete which is the automation framework which we use. My script on TeamCity always shows success irrespective of my tests failing or passing on the automation tool TestComplete. I want to add the test run info on the TeamCity build log tab, for eg if my test failed because of a bug and gives an error, I want to show the error on the build log. I tried using TeamCity service messages for this but it is not working as I would expect. Maybe I am missing something or doing it incorrectly. Please advise. All suggestions are welcome. Thank you!

This is how I am using the service messages. I am entering the messages in my .bat file which I am running from TeamCity.

##teamcity[testStarted name='SmokeTest1']
run SmokeTest1.bat
##teamcity[testFailed name='SmokeTest1' message='error']
##teamcity[testFinished name='SmokeTest1']

The above code runs my test and always says that my test failed based on my 'testFailed' command. What I want is that it should check if my test actually failed and report if failed (maybe it can check for some keywords to check if my test failed). But it seems like what is happening is that I am already telling it that the test has failed with the message 'error'. I wouldn't know before running the test if it is going to fail. Am I missing something here?

Also, in the below example, are we supposed to enter the error even before running the test? So, if I use testStdErr  and enter an error, it will always report that error irrespective of if that error really occurred during my test run?

##teamcity[testStarted name='SmokeTest1']

run SmokeTest1.bat

##teamcity[testStdOut name='testname' out='text']

##teamcity[testStdErr name='testname' out='error text']

##teamcity[testFinished name='SmokeTest1']

Thank you!

Regards,
Sam

4 comments
Comment actions Permalink

Hi Sam,

I'm not all that familiar with TestComplete, but maybe I can help you figure something out here. It looks like the success/failure of each test would be determined by the exit code. https://support.smartbear.com/testcomplete/docs/working-with/automating/command-line-and-exit-codes/exit-codes.html

Does your batch file return some information in either stdOut or stdErr that could be evaluated by the command line runner in TeamCity? What does it look like if you just run the batch file from a command line outside of TeamCity?

0
Comment actions Permalink

Hi Eric Borchardt! Thank you for your reply and apologies for my late reply. The exit codes from smartbear were certainly helped but it is still not working exactly as expected. Now, using the exit codes, it is returning an ECHO "There are errors" for an 'error' exit code. And then, using the TeamCity 'failure conditions', I am able to make the build fail if it finds the exact text "There are errors" in the build log which is only shown when there is an 'error' exit code. However, this way, my 'smoke test' build is shown as failed (which is great!), but my individual tests are all shown as passed, which is my concern. In the build log, it does show the "There are errors" text under individual tests, but that does not make my tests fail, rather only makes the smoke test build fail (which is also good and should happen). I am still puzzled as to how to make my smoke tests fail if there is an error "exit code".

Any ideas? Thank you!!

Here is a sample of my current code:

echo Running SmokeTest1..
echo ##teamcity[testStarted name='SmokeTest1']
run SmokeTest1 
echo Checking for errors or warnings using TestComplete exit codes..
call TestComplete_ExitCodes.bat
echo ##teamcity[testFinished name='SmokeTest1']

When I run the above block, it runs the test and if there is an error, it returns "There are errors" text in the build log. But the test still passes. I want to make the individual fail based on the exit code. How to do it?

0
Comment actions Permalink

Hi Sam,

I'm glad to hear you have partially scripted the interpretation of your TestComplete tests. It looks like you're just missing the testFailed service message that you originally had. Without it, I think the test will only be considered successful. I'm guessing you have already found this, but there is some information covering the use of test service messages in this section of our documentation: https://www.jetbrains.com/help/teamcity/build-script-interaction-with-teamcity.html#BuildScriptInteractionwithTeamCity-ReportingMessagesForBuildLog

I found this related YouTrack request for TestComplete integration. Please vote on this to show your support. We use our YouTrack site to determine which features and improvements are included in future releases of TeamCity. https://youtrack.jetbrains.com/issue/TW-18642

There is also a community created TeamCity/TestComplete integration walk-through that you may find helpful available here: https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206166669-TeamCity-and-TestComplete-Integration-a-Walkthough#5537415

0
Comment actions Permalink

Hi Eric Borchardt,

I was able to make my script work as expected! Thank you for sharing the link about the test service messages. Yes, I was playing with the service messages and trying to make them work for my script. I needed to add both - TestComplete exit codes and TeamCity testFailed service message and couple them using a conditional "If-Then" statement (as shown below). Now it is working as expected :)
Of course I can refine this further by adding additional exit codes for 'warnings', 'timeouts', 'success' using If-Else statements. I will be doing that next.

echo SmokeTest1..
echo ##teamcity[testStarted name='SmokeTest1']
run SmokeTest1
IF ERRORLEVEL 2 ECHO ##teamcity[testFailed name='SmokeTest1' message='This smoke test failed..' details='Check the html report under artifacts for details..']
echo ##teamcity[testFinished name='SmokeTest1']

Also thank you for sharing the YouTrack request! I have voted for this feature I am also following the related thread. I think this will be a very useful feature. Currently what I do is: In the TestComplete batch command we can add an argument to export the test results in HTML format. After export, I upload the HTML result on TeamCity as an 'Artifact'. So, after every test run, my HTML results are automatically zipped and uploaded as Artifacts on the TeamCity page for others to view and download. What is missing though is that the TestComplete test steps are not printed directly on the TeamCity build log as the test is running. Maybe it is a TestComplete limitation since I have seen it work (i.e. to display test steps on TeamCity build log) for at least one other automation framework.

Thank you for sharing the integration walk-through document. I had referred to this document when I had started integrating TeamCity with TestComplete.

Regards,
Sam

0

Please sign in to leave a comment.