What is the "<empty>" suite in Tests tab of build?

My build is using JUnit 3.8.1 and the Ant junit task to run a test suite. When the build completes, the Tests tab of the build shows a line with "<empty>" and another line with the actual test suite's name. If I switch it to the "tests" view, I see some lines with the correct test names but some lines are just empty except for the View test details icon.

My test suite uses a somewhat funky method of building up the suite of tests to run. It enumerates all the classes in a set of jars and finds all the TestCase-derived classes. Also, some of these TestCases are EnumeratedTestCases which enumerate over all the classes in a set of jars and perform bytecode analysis tests on each class. My suspicion is that these EnumeratedTestCases are not being detected properly by TeamCity's test results analyzer.

If I knew what could cause TeamCity to show "<empty>" for a test suite, maybe I could fix my TestCases and the test suite so that it can identify them correctly.

Thanks,
Gordon

21 comments
Comment actions Permalink

I just found this: http://www.jetbrains.net/devnet/message/5237947#5237947

My EnumeratedTestCase overrides toString() and returns the following:

getName() + ":" + getEnumeratedClassName() + "(" + getClass().getName() + ")"


Would this be confusing the test name parsing algorithm? If so, how can I add the enumerated class name to differentiate each invocation of the test so that it will be parsed correctly and presented neatly by TeamCity?

0
Comment actions Permalink

Hello Gordon,

  Currently, TeamCity can parse test names like

  package.a.b.c.ClassName.methodName(parameter1, parameter2)

  If you can convert your test names to such format, it should work.

  Hope, this helps,
  KIR

  P.S. We will try to improve parsing algorithm and to provide means to report various sections of test name independently

0
Comment actions Permalink

But I imagine it's also able to parse "testMethodName(package.a.b.c.TestClassName)" since that's the default toString of TestCase?

If so, what are the limitations on what kind of characters it accepts for each section of that string? I've tried various kinds of formats like "testMethodName_enumeratedClassName(TestClassName)" with the enumeratedClassName's dots replaced with underscores and it still wouldn't parse.

I'll try constructing it like you suggested, "TestClassName.testMethodName(enumeratedClassName)", and see if that works.

Thanks,
Gordon

0
Comment actions Permalink

I just tried:

getClass().getName() + "." + getName() + "(" + getEnumeratedClassName() + ")"


And TeamCity still wouldn't parse it. Is there any way I can debug this? These tests form a major percentage of my test suite and I would really like to see TeamCity's reporting on them.

Thanks,
Gordon

0
Comment actions Permalink

After some further investigation it appears that it is parsing the name, but it seems to be ignore the parameter part of the name and after 100 tests with the same name (ignoring parameter), the name on subsequent tests from the same test case are blank.

For example, I have an enumerated test case called TestAllNotificationListenerExceptionHandlers which contains a test method called testNotificationListenerExceptionHandlers. From my own logging, I have determined that it runs 3694 times, i.e. it enumerates over 3694 classes and runs the test method on each class. In the TeamCity build log, if I go to the Tests view and sort by Order#, I find that TestAllNotificationListenerExceptionHandlers tests start at order# 108 but the test from order# 209 onwards have no name.

0
Comment actions Permalink

Hello Gordon,

  The problem is, TeamCity cannot handle correctly more than 100 tests with exactly same name.

  Could you please try to wrap your parameters in "" in the generated test name? I hope this will allow TC to recognize parameters.

  If not, could you probably attach a snippet from your Tests page (or from build log) so we could take a look at how your test names look like?

  Regards,
  KIR

0
Comment actions Permalink

I'll try wrapping the parameter in quotes. In my last attempt, the Test.toString() looked like this:

com.sitraka.pas.TestAllNotificationListenerExceptionHandlers.testNotificationListenerExceptionHandlers(com.sitraka.pas.agent.Agent)

Thanks,
Gordon

0
Comment actions Permalink

The quotes didn't make any difference.

In the build log, before the test names become blank, they look like this:

TestAllNotificationListenerExceptionHandlers.testNotificationListenerExceptionHandlers (com.sitraka.pas.AutoTestAll:com.sitraka.pas)

Thanks,
Gordon

0
Comment actions Permalink

Hello,

   Another thing to try is to prepend some package before class name. I.e. generate something like:
   root.package.TestAllNotificationListenerExceptionHandlers.testNotificationListenerExceptionHandlers(com.sitraka.pas.AutoTestAll:com.sitraka.pas)

   for a case when real package is empty.

   Hope this helps,

   KIR

0
Comment actions Permalink

They always have a package name. That last output I showed you was from the TeamCity webpage, a line from the lists of tests with the various scoping links. My previous messages shows the toString output includes the test class's package.

However, other tests, which are not EnumeratedTestCases, use the standard TestCase toString format: testMethodName(package.TestClassName). Would the use of two different formats in the same suite of tests be the problem?

0
Comment actions Permalink

Hello Gordon,

  Sorry for the delay with response.
  Could you please watch issue TW-8012. I'm going to fix the problem discussed in this thread as a part of this issue.

  Thanks!
  KIR

0
Comment actions Permalink

That will make it at least possible to see their stats, but won't that mean that they are still segrated from the first 100 tests of the same kind which were named correctly? I would prefer that all the tests are properly named.

0
Comment actions Permalink

gtyler wrote:

That will make it at least possible to see their stats, but won't that mean that they are still segrated from the first 100 tests of the same kind which were named correctly? I would prefer that all the tests are properly named.


   I'd really like to fix naming part as well. I'd really appreciate if you create a sample project which generates invalid test names and attach it to the issue. But I'll try to figure out the naming problem anyway.
   Thanks!

   KIR

0
Comment actions Permalink

So you're saying that if you can fix the parsing so that it parses each test name as a unique name, it will indirectly solve the problem of the 100 same-named tests limit?

0
Comment actions Permalink

I believe you don't really have > 100 exactly same-named tests. I think that as soon as TeamCity will recognize the tests as different ones (for instance, because of different parameters), the problem will be solved.

0
Comment actions Permalink

Okay, well here is a small project with a TestCase whose suite method builds a TestSuite of 500 test, each of which have a parameter set after standard instantiation through new TestSuite(TestMyApp.class). TestMyApp has had its toString overridden to produce a name in the format "package.Class.testName(param)". I haven't tested it in TeamCity because I don't want to check it into my company's CVS, but the test suite runs in IDEA and it thinks they're all called the same thing even though the actual test method prints the toString() to stdout and proves that each instance of the test is named differently.



Attachment(s):
UnitTestNameParsingBug.zip
0
Comment actions Permalink

Hello Gordon,

   Looks like the we do not consult toString to generate test name from TestCase class. That's why it doesn't work for you.
   I'll try to push the fix to TeamCity 4.5.2, which is about to be released on this week (or in the beginning of the next one).

   Kind regards,
   KIR

0
Comment actions Permalink

So you use the actual class name of the TestCase and the getName()? I can try changing that and see what effect it has.

0
Comment actions Permalink

I tried that, unfortunately after call like setName("testMethod(param)") test could not be run anymore

0
Comment actions Permalink

However, it appears that if I just override getName() in my TestCase, it runs and IDEA names it correctly (including parameter) in its test runner toolwindow. I tested this with the test app I just posted, but I'll try it now with our real project's tests.

0
Comment actions Permalink

This did the trick. When I override getName() to return:

super.getName() + "(" + getEnumeratedClassName() + ")"


It is parsed correctly in TeamCity and I can see all my tests now.

Thanks!

0

Please sign in to leave a comment.