Tests with equal names in different test suites

Hi

I'm using your test format messages for resource testing in our project.
Output looks like

##teamcity\[testSuiteStarted name='house1.res'\]
##teamcity\[testStarted name='Texture_Size (Texture1.tga)'\]
##teamcity\[testFailed name='Texture_Size (Texture1.tga)' message='Big Size' details='stack trace'\]
##teamcity\[testStdErr name='Texture_Size (Texture1.tga)' out='texture SizeX SizeY'\]
##teamcity\[testFinished name='Texture_Size (Texture1.tga)'\]
##teamcity\[testStarted name='Texture_Size (Texture2.tga)'\]
##teamcity\[testFinished name='Texture_Size (Texture2.tga)\']
##teamcity\[testStarted name='Texture_Size (Texture3.tga)'\]
##teamcity\[testFinished name='Texture_Size (Texture3.tga)'\]
##teamcity\[testStarted name='Triangles_Count (house1.res)'\]
##teamcity\[testFinished name='Triangles_Count (house1.res)'\]
.....................
.....................
.....................
##teamcity\[testSuiteFinished name='house1.res'\]

##teamcity\[testSuiteStarted name='house2.res'\]
##teamcity\[testStarted name='Texture_Size (Texture1.tga)'\]
##teamcity\[testFailed name='Texture_Size (Texture1.tga)' message='Big Size' details='stack trace'\]
##teamcity\[testStdErr name='Texture_Size (Texture1.tga)' out='texture SizeX SizeY'\]
##teamcity\[testFinished name='Texture_Size (Texture1.tga)'\]
##teamcity\[testStarted name='Texture_Size (Texture4.tga)'\]
##teamcity\[testFinished name='Texture_Size (Texture4.tga)'\]
##teamcity\[testStarted name='Texture_Size (Texture5.tga)'\]
##teamcity\[testFinished name='Texture_Size (Texture5.tga)'\]
##teamcity\[testStarted name='Triangles_Count (house2.res)'\]
##teamcity\[testFinished name='Triangles_Count (house2.res)'\]
.....................
.....................
.....................
##teamcity\[testSuiteFinished name='house2.res'\]

I run checker twice on same resources and on the second path it found new errors. All new errors was about texture sizes.
I think it's because some textures are reused in many houses, like texture1 in my sample. So we had test with equal names in different test suites.

Is it my mistake and I had to change checker in the way to make all test names unique, or shall write an isssue in your tracker?

--
Best regards,

Alexander Shishenin
QA Lead at Client Team
Moscow Development Studio
Nival Online

6 comments

Alexander,

Usually, test name should include suite's name in it. This should make all the test's names different.

Having several tests with the same name is not a good idea, though, TeamCity should handle them as different tests and provided the tests are always run in the same order this should not be a problem.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Yegor,

I'll already made test names unique.
My question - Is it "Function As Designed" or I can write issue in your tracker?


--
Best regards,

Alexander Shishenin
QA Lead at Client Team
Moscow Development Studio
Nival Online
http://www.nivalonline.com

0

Alexander,

Can you please describe the exact problem you have with the same-named tests?

What do you expect and what do you get?

As I described, TeamCity should cope with that OK, provided the order of the tests is not changed from build to build.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Yegor,

I was expecting that test names had to be unique just for one test suite.
But TeamCity each run found new failed tests.

For example.
##teamcity\[testStarted name='Texture_Size (Texture1.tga)'\]
##teamcity\[testFailed name='Texture_Size (Texture1.tga)' message='Big Size' details='stack trace'\]
##teamcity\[testStdErr name='Texture_Size (Texture1.tga)' out='texture SizeX SizeY'\]
##teamcity\[testFinished name='Texture_Size (Texture1.tga)'\]

Such test output in different test suites will be marked as new fail test each run.
Full example log was written in the first message of this thread.

I can send you full logs via e-mail. It's size 17Mb.

--
Best regards,

Alexander Shishenin
QA Lead at Client Team
Moscow Development Studio
Nival Online
http://www.nivalonline.com

Edited by: Alexander Shishenin on Jun 24, 2008 11:06 AM

Edited by: Alexander Shishenin on Jun 24, 2008 11:29 AM



Attachment(s):
TeamCity.bmp
0

Alexander,

It seems I've got your problem: you get the service messages-reported tests marked as "new" in every build.

I cannot reproduce it with the snippet you provided in the first message, the tests are not marked new in my case (at least on the current sources).

I guess it's time for the issue in the tracker.
Can you please try to narrow the test data that reproduces the problem and post it into the issue?
Please also note your TeamCity version in the issue.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Yegor,

I create issue 5330.
Full builder log you can find in attache arhive.

--
Best regards,

Alexander Shishenin
QA Lead at Client Team
Moscow Development Studio
Nival Online
http://www.nivalonline.com

0

Please sign in to leave a comment.