Ant JUnit report watcher reporting incorrect test durations

Hi,

I'm using TC 4.5.4 and antunit 1.1.

I run 6 tests each lasting about 200 seconds. My first test case's duration shows up correctly, but the remaining 5 show <1 ms.

Here's my build output:

[au:antunit] Target: testIntelRelease-MOTW-KJFK took 213.328 sec
[15:15:29]: [au:antunit] Target: testMicrosoftRelease-COTW-KJFK took 211.641 sec
[15:15:29]: [au:antunit] Target: testMicrosoftRelease-MIF-KJFK took 211.703 sec
[15:15:29]: [au:antunit] Target: testMicrosoftRelease-MOTW-KJFK took 211.688 sec
[15:15:29]: [au:antunit] Target: testIntelRelease-MIF-KJFK took 211.735 sec
[15:15:29]: [au:antunit] Target: testIntelRelease-COTW-KJFK took 211.703 sec

Here is a snippet of my xml report:

<testsuite buildFile="D:\Apps\BuildAgent\work\ee894e159a7014ac\tests.xml" hostname="AUTO-TEST2" name="tests_xml" timestamp="2009-09-08T18:54:13">
  <properties>
  </properties>
  <testcase name="testIntelRelease-MOTW-KJFK">
    <time>213.328</time>
  </testcase>
  <testcase name="testMicrosoftRelease-COTW-KJFK">
    <time>211.641</time>
  </testcase>
  <testcase name="testMicrosoftRelease-MIF-KJFK">
    <time>211.703</time>
  </testcase>
  <testcase name="testMicrosoftRelease-MOTW-KJFK">
    <time>211.688</time>
  </testcase>
  <testcase name="testIntelRelease-MIF-KJFK">
    <time>211.735</time>
  </testcase>
  <testcase name="testIntelRelease-COTW-KJFK">
    <time>211.703</time>
  </testcase>
  <tests>6</tests>
  <failures>0</failures>
  <errors>0</errors>
  <time>1275.797</time>
</testsuite>


Any help would be greatly appreciated.

Thanks,

Scott
0
21 comments
Avatar
Permanently deleted user

Hi, Scott,

Could you, please, update xml-report-plugin and make sure that this problem still occurs. If it really occurs, please, attach full build log.

0
Avatar
Permanently deleted user

Hi,

I updated to the latest version. Restarted the server.

Also made sure that agent and server times were in sync.

The problem still exists.

Attached is the full log and the xml report.

Thanks,

Scott



Attachment(s):
RTS_AutoTests.log.zip
TEST-tests_xml.xml
0
Avatar
Permanently deleted user

Thanks a lot for the attachments.

One more thing we need to localize the problem: could you, please, turn agent log to debug mode (go to <build agent installation path>\conf\teamcity-agent-log4j.xml and remove lines that are marked with "DELETE THIS LINE FOR ENABLING DEBUG LOGGING" text), run the build and provide here the agent log file (is at <build agent installation path>\logs\teamcity-agent.log). Agent restart before running build is not necessary.

0
Avatar
Permanently deleted user

Hi.

Attached is the agent log...

Another observation...If a test failed, the time is correct. otherwise, the first test is the only correct time reported.

Scott



Attachment(s):
teamcity-agent.zip
0
Avatar
Permanently deleted user

Hi,

Any progress on this issue?

Also, I've attached another XML report which has failures with stacktraces. The stacktraces do not show up in TC.

Thanks,

Scott



Attachment(s):
TEST-performance-tests_xml.xml
0
Avatar
Permanently deleted user

Thanks for information and attachment!

This problems occur since AntUnit produces xml reports with structure that differs from Ant junit task xml reports schema. It will be fixed in the nearest time by supporting this new format.

0
Avatar
Permanently deleted user

Alternatively, is there a fix I could do to XMLAntUnitListener.java in AntUnit?

I see that the actual ANT builds here -- http://teamcity.jetbrains.com/viewType.html?tab=buildTypeStatusDiv&buildTypeId=bt133 -- use AntUnit and their test results show up correctly.

Thanks,

Scott

0
Avatar
Permanently deleted user

The originally supported format was:

<!ELEMENT testsuites (testsuite*)>

<!ELEMENT testsuite (properties, testcase*,
                  failure?, error?,
                   system-out?, system-err?)>
<!ATTLIST testsuite name      CDATA #REQUIRED>
<!ATTLIST testsuite tests     CDATA #REQUIRED>
<!ATTLIST testsuite failures  CDATA #REQUIRED>
<!ATTLIST testsuite errors    CDATA #REQUIRED>
<!ATTLIST testsuite time      CDATA #REQUIRED>
<!ATTLIST testsuite package   CDATA #IMPLIED>
<!ATTLIST testsuite id        CDATA #IMPLIED>



<!ELEMENT properties (property*)>


<!ELEMENT property EMPTY>
<!ATTLIST property name  CDATA #REQUIRED>
<!ATTLIST property value CDATA #REQUIRED>


<!ELEMENT testcase (failure?, error?, system-out?, system-err?)>
<!ATTLIST testcase name       CDATA #REQUIRED>
<!ATTLIST testcase classname  CDATA #IMPLIED>
<!ATTLIST testcase time       CDATA #REQUIRED>


<!ELEMENT failure (#PCDATA)>
<!ATTLIST failure message CDATA #IMPLIED>
<!ATTLIST failure type    CDATA #REQUIRED>


<!ELEMENT error (#PCDATA)>
<!ATTLIST error message CDATA #IMPLIED>
<!ATTLIST error type    CDATA #REQUIRED>


<!ELEMENT system-err (#PCDATA)>

<!ELEMENT system-out (#PCDATA)>

AntUnit reports were supposed to fit it, but actually in many cases they don't.
But still I can't get any AntUnit xml repots schema at public resources.

One more source of possible report data skipping is the "parsing on the fly".
May be you can observe how do reports appear on disk during the build? Are they published once at the end of all test case or partly after each test is finished?

0
Avatar
Permanently deleted user

One more source of possible report data skipping is the "parsing on the fly".

>> I can try to disable parsing via the build config setup page.

May be you can observe how do reports appear on disk during the build? Are they published once at the end of all test case or partly after each test is finished?



>> I've observed that the XML is published partly during and after each test...
>> I'll give it a try right now and let you know.

Scott
0
Avatar
Permanently deleted user

Some good news!

- By disabling real-time parsing of XML reports AND enabling post test parsing via service messages, I now get the correct test times.

The bad news:

- Tests with errors do not show stacktraces...

I've attached another XML report.

Thanks for your help!

Scott



Attachment(s):
TEST-simple-smoke-tests_xml.xml
0
Avatar
Permanently deleted user

Hi Victory,

Any news on this problem? Do you have any beta code you want me to try?

Thanks,
Scott

0
Avatar
Permanently deleted user

Hi,

I'm getting another error too.

D:\Apps\BuildAgent\work\6.4smoke\xml-test-reports\TEST-simple-smoke-tests_xml.xml: failed to parse with Ant JUnit parser

The file is attached.

Thanks

Scott



Attachment(s):
TEST-simple-smoke-tests_xml.xml
0
Avatar
Permanently deleted user
0
Avatar
Permanently deleted user

Hi, Scott,

Excuse me for my silence.

TEST-simple-smoke-tests_xml.xml seems to be well-stuctured. But this error tells that at the end of the build the report is still unfinished or wrong-formated.
Do you get it regularly?

May be you can attach any snapshots of a report as it is at some stage of testing (after several tests finish)?

0
Avatar
Permanently deleted user

I uploaded the zip. do I need to restart the server?

I re-ran it and it does not show the stacktrace still, nor the exception message.

thanks,

Scott



Attachment(s):
TEST-simple-smoke-tests_xml.xml
0
Avatar
Permanently deleted user

To install the plugin put xml-report-plugin.zip into <TeamCity Data Directory>/plugins folder and restart the server.
You should also see that after the server restart agents start upgrading.

0
Avatar
Permanently deleted user

>> I'm getting another error too.

>> D:\Apps\BuildAgent\work\6.4smoke\xml-test-reports\TEST-simple-smoke-tests_xml.xml: failed to parse with Ant JUnit parser

Please look attentively at the TEST-simple-smoke-tests_xml.xml (I attach it again for the convenience ) - there is no closing tag for last <testcase> element. Error is reported due to wrong XML structure of the report.



Attachment(s):
TEST-simple-smoke-tests_xml.xml
0
Avatar
Permanently deleted user

Hi.
I still cannot see my stacktraces or any error messages even though my tests have failed.
Any ideas?
Thanks,
Scott

0
Avatar
Permanently deleted user

Hi, Scott.

Thanks for attention to this problem!

It's a bit strange that you still can not see error stacktraces, these bug was fixed.
I'l try to investigate your problem once more.

What TeamCity version are you using now?
Are you sure plugin version is up-to-date? (Suggest you http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt143&buildId=lastSuccessful if you use 4.5.5)
Did you notice my previous post about TEST-simple-smoke-tests_xml.xml?
Besides, are test durations shown correctly?

0
Avatar
Permanently deleted user

Hi,

I'm using TC 4.5.4. And yes, I took the latest build from your TC instance. I've restarted the server since I updated the plugin. Is there a way to confirm the version of the plugin that is running?

My durations are correct now. I no longer have my runner configured to watch for the xml file. Instead I only issue an importData service message once all tests are done.

I've re-attached my latest xml file for you to look at.

Thanks,

Scott



Attachment(s):
TEST-simple-smoke-tests_xml.xml
0
Avatar
Permanently deleted user

Thanks for attachments!

To make sure that new plugin version is loaded please examine your teamcity-server.log (at <TeamCity home\log>) for warning like this:

Plugin xml-report-plugin[version: ... path: ...\xml-report-plugin] will not be loaded because it is overridden by plugin: xml-report-plugin[version: ... path: ...\xml-report-plugin]

It should make clear what plugin version is working.
Copy this warning here, please.
It looks as if you have old one.

0

Please sign in to leave a comment.