How does TeamCity interpret the TAP message description?

Hello,

As I don't use any unit testing framework in my C++ project, I opted for printing the unit test message in TAP format. I installed a plugin to parse TAP messages in the build output from here (many thanks to javascript:; for the nice work!):
https://github.com/pavelsher/teamcity-tap-parser

The spec for this format is quite simple:

1..N
ok     1 Description of test 1
not ok 2 Description of test 2
...
ok     N Description of test N


Long story short, I noticed (by chance), that, if the description has a certain format, TeamCity interprets it differently. For example, if the format is like this:

ok 1 Test File: file open
ok 2 Test File: file read
ok 3 Test File: file write
ok 4 Test File: file close


I noticed that TeamCity can group these 4 tests by scope, the scope being "Test File". Check out the image below:
test-scope.png
The question is: is there a specification for the message format that TeamCity expects / can parse, so that I can fully make use of its grouping / categorizing features?

Thanks!

4 comments
Comment actions Permalink

Well, such grouping is not a plugin feature. This is a feature of TeamCity itself. If test name starts with prefix: text, TeamCity treats prefix as a suite name and then groups tests by this name.
As for the supported TAP features, as a more or less formal grammar I used this one:
http://search.cpan.org/dist/Test-Harness/lib/TAP/Parser/Grammar.pm#TAP_GRAMMAR

Not all of this is supported as I remember. And there is an issue with stderr / stdout. If test ouputs something to stderr, this stderr may not be synced with the line about the test in the TeamCity build log. I.e. the test line may be printed long before or after the test stderr. This seems to be a fundamental problem of how stderr/stdout streams work in Java.

Anyway, thanks for using this plugin, feel free to send me your feedback!

0
Comment actions Permalink

Well, such grouping is not a plugin feature. This is a feature of TeamCity itself. If test name starts with prefix: text, TeamCity treats prefix as a suite name and then groups tests by this name.

Indeed, but I figured there is more to this feature than this rule. This is what I want to find out.

As for the supported TAP features, as a more or less formal grammar I used this one:
http://search.cpan.org/dist/Test-Harness/lib/TAP/Parser/Grammar.pm#TAP_GRAMMAR

Not all of this is supported as I remember. And there is an issue with stderr / stdout. If test ouputs something to stderr, this stderr may not be synced with the line about the test in the TeamCity build log. I.e. the test line may be printed long before or after the test stderr. This seems to be a fundamental problem of how stderr/stdout streams work in Java.

Anyway, thanks for using this plugin, feel free to send me your feedback!

Ok, I sure will!

0
Comment actions Permalink

Ok, so no one can point me to some formal way TeamCity parses test messages?

0
Comment actions Permalink

TeamCity expects test name to look like:
<test suite>: <package>.<class>.<test method>
<test suite> is just an arbitrary string which does not contain colon
<package> consists of names separated by dots.
<class> is a name without dots
<test method> is the rest of the reported line

This format mostly suits .NET and Java. TeamCity itself does not know much about test and just tries to parse test name with this format. If it can't the whole line is treated as test name. Otherwise, if parsing was successful, TeamCity will be able to group tests by various parts (package, class, suite).

0

Please sign in to leave a comment.