Problems with restarted JUnit tests

Hello to all TeamCity support!
I've wrote you about problem with restarted tests:
http://devnet.jetbrains.com/thread/455712?tstart=0

Some test may run twice: first time as "failed" and second time as "restarted - successful".
What  is the problem is: teamcity cant "fix" first, "failed" test, and, in  tests report i see two same tests, failed and successful.
Can i "fix" those failed tests somehow, using pligin OpenApi?

I've found STestRun interface, but, it does not allow to mute/investigate this test.
Is it possible anyway?

We installed latest Teamcity 9 EAP, and there all the same tests (failed and non-failed) are grouped in one test.
That's cool, but we still miss "auto-fixing" feature, when second, successful run of test will "fix" first, failed test (the same test).
failed_test_is_not_fixed.png

As you can see on this image, the secord, successful run of test, does not "fix" the first, failed run.
Can you implement this "auto-fixing" feature in next TC9 EAP?

If not, can i implement this feature using OpenApi? STestRun interface is read-only, i can't mute/investigate test using it.

6 comments
Comment actions Permalink

Very important problem! Please help!!


0
Comment actions Permalink

Hello,

   I don't think we'll implement a logic when a successful test run in the same build will fix the previous failure.

   This will contradict with the current behaviour when invocationCount is set to detect flacky tests. In this situation, any failure means that the test fails.

   I think that if you have such tests, you should mute them so they don't break your builds, and make them more stable afterwards. Would it work?

   Regards,
   KIR

0
Comment actions Permalink

Hi Kirill!

maxkir wrote:

...

   I think that if you have such tests, you should mute them so they don't break your builds, and make them more stable afterwards. Would it work?

Yes, i can mute this myself, but, i want to do it automatically.
Can i do it using some plugin API?

I've found STestRun interface, but, it does not allow to mute/investigate tests, it seems to be read-only interface to saved test.
Is it possible to change test status using OpenAPI interfaces?

0
Comment actions Permalink

Please ask: Is it possible to change test status using OpenAPI interfaces?
If this is not possible as Teamcity feature, then, can i do it myself using OpenAPI interfaces?

0
Comment actions Permalink

Please ask: Is it possible to change test status using OpenAPI interfaces?
If this is not possible as Teamcity feature, then, can i do it myself using OpenAPI interfaces?

0

Please sign in to leave a comment.