Running tests concurrently in TeamCity

Hi all,

I'm running several test concurrently as part of my build. TeamCity did not have built-in support for the testing framework that I'm using (Fit) so I'm using service messages to send test information to TeamCity. However, because tests are run concurrently, "testStarted" and "testFinished" messages associated with a certain test are usually overlapping and/or nested within messages of other tests. For example:

##teamcity[testSTarted name='Test1']
##teamcity[testSTarted name='Test2']
##teamcity[testSTarted name='Test3']
##teamcity[testSFinished name='Test2']
##teamcity[testSFinished name='Test1']
##teamcity[testSFinished name='Test3']

Although "testStarted" and "testFinished" messages are printed for all tests, TeamCity is only detecting some of the tests (about half, the number changes in different runs).
Is TeamCity capable of detecting the start and finish of concurrent tests? If so, what am I doing wrong here and what do I need to do to get to work.



Comment actions Permalink

Hello, Mayad

You should use flows. A flow in TeamCity build log messaging is analogous to a thread in execution.
Generally all messages that must come in specific order should be posted in the same flow.
A flow is represented with a unique string id. All you have to is provide additional attribute 'flowId' in your service messages.

Your example would look like this:

##teamcity[testStarted name='Test1' flowId='aaaa']
##teamcity[testStarted name='Test2'  flowId='bbbb']
##teamcity[testStarted name='Test3'  flowId='cccc']
##teamcity[testFinished name='Test2'  flowId='bbbb']
##teamcity[testFinished name='Test1'  flowId='aaaa']
##teamcity[testFinished name='Test3' flowId='cccc']

You can make a direct mapping from your test threads to flows: one thread - one flow. You can even post each test in a separate flow but this would make the server consume much more resources since flows are closed in the very end of the build. In case of big number of tests this could cause server instability. So the best strategy is having minimum possible flows.

Comment actions Permalink

Exactly what I was looking for! Thank you :-)

Comment actions Permalink

Hi Sergey!

A follow up on Mayad's questions: while having multiple tests running in parallel in separate threads/processes, Teamcity will get the stdout and stderr from the tests all mixed up together. Is there a way to tell Teamcity that this particular line on stdout belongs to that particular test?

Thanks in advance!


Please sign in to leave a comment.