Random test failure due to multiple console writers
In our company we have TeamCity on premise - TeamCity Enterprise 2020.1 (build 78475)
We saw weird behavior about number of tests changing on same source codes. After investigation we found that while executing ‘dotnet test’ there are some random problems and this can give different number of test depending on when some test steps on each other’s.
On side note for this topic – to analyze this problem I saw that new test UI is poor. Only when switch to classic view I could group by project/namespace and do some compare between them.
After finding missing tests I saw two kind of parsing problem in logs:
- "while parsing TeamCity service message : Value should end with "'". Valid service message has a form of"
- "The service message is invalid because it does not end with ]"
For example:
[09:07:43] : [dotnet test] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[11]
[09:07:43] : [dotnet test] List of registered output formatters, in the following order: Microsoft.AspNetCore.Mvc.Formatters.HttpNoContentOutputFormatter, Microsoft.AspNetCore.Mvc.Formatt[xUnit.net 00:00:01.32]
[09:07:43] : [dotnet test] ##teamcity[testFinished name='(....), deers.StringOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StreamOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter
[09:07:46]W: [dotnet test] The service message is invalid because it does not end with ] (...)
[09:07:43] : [dotnet test] dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[4]
Problems is in integration tests because in those tests there is normal ASP.NET Core app – we are mocking external stuff but full ASP pipe is loaded and on default settings MS logs are landing on console.TC is using console for internal stuff with ##teamcity and that is the problem.
Of course we can try to disable console in CI tests but this behavior look very fragile. Potentially we can have missing some test due to someone writing to console.
Is there something you are proposing for this problem?
Please sign in to leave a comment.
Hi Dominik,
This is a known issue. Please follow this bug to get notified about when it is fixed: https://youtrack.jetbrains.com/issue/TW-62349.
Regarding the new tests UI, please be informed it is still in development and we plan to implement the missing features in the future versions: https://youtrack.jetbrains.com/issue/TW-62537. Thanks for your feedback though.
-Anatoly
Register for TeamCity Technology Day - Nov 5th