Test counts are wrong, possibly due to qualified name of tests being wrong
My issue looks to be very similar to this issue:
I'm using Quick BDD for writing tests in an iOS project.
I can see from the build log that TC's iOS agent is running all of my tests ok. But the web interface reports a wrong number each time (230, then 227, then 229, etc)
When I download the CSV of various test sessions, I can see that any number of specs between 0 and n are being reported for each row. For example (sanitised example):
All tests: AppNameTests.xctest: MySpecA: MySpecB: test_name_from_spec_b
Clearly there shouldn't be two specs listed in one line.
Other times, there is no spec listed at all:
All tests: AppNameTests.xctest: test_name
This might explain the incorrect test counts reported by TC web interface.
This feels like a concurrency issue.
Some of my tests are async in nature; they use Quick's mechanisms to wait on something happening. Could that be throwing TC off?
My TC setup is quite simple. I have a simple time-based build trigger that builds upon changes in VCS. It runs 'build test'.
Please sign in to leave a comment.
Hi Alex,
thanks for your report but there is a small issue here at hand: We don't know how your build reports the information about the tests. We don't have embedded support for "Quick BDD". It's possible that you are running them through another supported runner such as XCode, you are using the XML Report feature or maybe posting them via service messages. Without that information, we don't know what could cause the issue.
If you are using service messages and suspect that concurrency could be the issue, we introduced "flowId" a while ago: https://www.jetbrains.com/help/teamcity/build-script-interaction-with-teamcity.html#Message+FlowId . This feature allows reporting the messages in a more granular way that should make it easier for teamcity to track.