Test count discrepancy when muting/investigating

Hi All -

We are just starting to use this feature and have hopes that it will be increadibly helpful to manage a large body of tests. We run the same set of Jasmine tests on many browsers and so we "wrap" them in a top-level suite named by the browser/version. This seems to work and bring in all the tests. But we are running into problems when we have failures that we want to mute/investigate. I see one number for the total number of failures, but when I click "All" and then the Investiage/Mute... link, I get a much smaller count. I suspect this has to do with leaf-names being duplicated, but that is just a theory.

Are there any limitations around this kind of thing? We could try to work around such a limitation by "tagging" the leaf-names to make them consistent and unique, but that would be some work. It could be an issue on our side and we are still digging in to it, but I wanted to ask if this is any kind of known issue before spending too much more time on it.

Thanks in advance! Keep up the great work - love TC :)

@dongryphon

3 comments
Comment actions Permalink

Hi,

This may be a bug (or may be not). It would be very helpful if you provide additional details:
- TeamCity version and build number;
- the screenshot of the page with the tests list;
- the screenshot of the "Investigate/Mute" dialog (if I understand you right, that is where you get confused).

I think the better place to continue a discussion is our issue tracker: http://youtrack.jetbrains.com/issues/TW
Could you please file an issue? You can restrict the permissions for your screenshots, if you're concerned about privacy.


--
Maxim

0
Comment actions Permalink

Hi Maxim,

Thanks for the reply.

I believe the problem is in our data gathering and feed into TeamCity. I just wanted to understand how TC identifies tests from the log. Since we use nested Jasmine suites around the final spec (test), we are concerned that TC might use the spec name (the leaf name) of this nested structure to identify the test.

At least that is one possible explanation for the discrepancy between the list view of all failures and the count displayed in the dialog when All are selected.

It is also possible we have problems in our aggregation of test runs from all the various browsers. In total, we have over 100k specs to feed in to TC (true number is this divided by number of browsers, which is large :)). Anyway, if TC uses the full "pah" (including suites) to identify tests in the log, then we will keep digging to see if we are somehow duplicating chunks. If TC uses only the leaf name (the spec), then we can decorate that as needed.

Once we believe our feed is really solid and totally distinct (the definition of this really depends on the answer to this question) we will file an issue and proceed down that road.

Thanks again!

Best,
Don

0
Comment actions Permalink

Concerning correct test reporting please consult with this document: http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity


--
Maxim

0

Please sign in to leave a comment.