Service messages improvement

Answered
Avatar
Nikolay
Created

Hello,

I'm developing a new version of the Siesta testing tool: https://www.bryntum.com/products/siesta/

In the current version we had some poor man TeamCity reporter and I'm now improving it. 

I've noticed, that service messages implementation is sub-optimal currently. I'd like to share my thoughts how it can be improved.

- First of all, it uses the "test" concept instead of "assertion", which is a bit weird. When one says "test" s/he probably means a test file, or, sometimes a group (it/describe section) of assertions. In other words, "test" is a "parent" tree node in the tests results. Assertion is a "leaf" tree node, containing a small, atomic statement about the code flow. Well, I can live with it, it is just a bit uncommon terminology.

- Second, it misses the concept of "test log message". Sometimes in the test, you'd like to log some information in the output report, for example for debugging purposes. Currently the only way to output such information is another `testStarted/testFinished` pair.

- Then I'd also use a separate entity for the "exception", as it is something we'd like to be instantly noticeable in the report.

- Current implementation is also missing any capability to style the output (color / background-color at least). It would be great if it would support the ANSI terminal styling codes (as in `chalk` nodejs module)

In general test results should be organized as tree (or several trees with different `flowId`). The "parent" nodes in the tree are "tests", the "leaf" nodes are assertions/log messages/exceptions.

Hope all this makes sense.

Please consider these improvements.

Thank you, Nickolay

Votes

1

Share

6 comments
Avatar
Nikolay
Created

Do you think these improvement suggestions are reasonable?

0
Avatar
Anatoly Cherenkov
Created

Hi Nickolay,

Test log message can be sent using the ##teamcity[message flowid='<id>' text='<text>'] service message. Example:

echo "##teamcity[testStarted flowId='0' name='test.A']"
echo "##teamcity[message flowId='0' text='some message about test.A']"
echo "##teamcity[testFinished flowId='0' name='test.A']"

For passing expectations, you can use the message attribute of the testFailed service message. Example:

echo "##teamcity[testStarted flowId='0' name='test.A']"
echo "##teamcity[testFailed flowId='0' name='test.A' message='Some exception']"
echo "##teamcity[testFinished flowId='0' name='test.A']"

The message would be highlighted in the build log and shown on the Tests tab of the build results page. Would that suffice?

For coloring build log please vote for this feature: https://youtrack.jetbrains.com/issue/TW-5724.

Regarding organizing the test results in a tree, if this feature request gets implemented, it should cover your use case: https://youtrack.jetbrains.com/issue/TW-29043. Please vote/watch.

-Anatoly 

0
Avatar
Nikolay
Created

Hi Anatoly,

Thanks for the reply for pointers to those issues. I'll try with the 'blockOpened/Closed' message as described in comments for https://youtrack.jetbrains.com/issue/TW-29043

The coloring ticket seems to be very old, but it should improve the "ergonomics" a lot. Ideally should just support the ANSI control sequences, to not implement the different coloring scheme for TeamCity.

Not sure I get your idea about including the passing expectation in the "testFailed" message, this seem a bit contradicting. I'm just pointing that the data model of the services messages is sub optimal, as well as terminology. It seems to be heavily tied to the Java testing conventions.

Check this link https://www.jetbrains.com/help/teamcity/2020.2/service-messages.html#Interpreting+test+names for example. It says about suite name, package, class name, test method and test parameters. Only the "suite name" term is readily recognized by people with non-Java background. In other languages, like JavaScript/TypeScript testing terminology operates on "test suite, test file, "it/describe" section, and assertion/expectation.

I believe the service messages mechanism needs to be revamped from scratch, under the different name, like "logging messages". It should operate on more abstract level, not tied to conventions in any particular languages. The tree structure I'm proposing seems to be generic enough to cover all cases.

0
Avatar
Permanently deleted user
Created

Oddly enough, I just pushed a new PowerShell Module that is a Service Message Helper, based on the documentation you referenced.

I wrote this years and years ago, and thought it was finally time to share it. It's by no means official, or really supported -- but it will run on any version of PowerShell you have. Please check it out. I'm curious what others think.

https://github.com/edhaack/teamcity-service-messages

0
Avatar
Permanently deleted user
Created

Opps! I forgot to include this to my last message:

I know this isn't related specifically to Service Messages for Tests and/or Coloring -- but the module does help eliminate the need for remembering the exact syntax along with verifying that the message is parsed correctly.

0
Avatar
Nikolay
Created

Thanks for sharing Ed!

0

Please sign in to leave a comment.