Why do muted tests that pass automatically become muted again when they next fail?

I am testing the test muting feature before making general use of it.

  1. I created 3 failing tests, ran a build, then muted the failing tests, with "unmute" option set to "when test passes successfully".
  2. I ran another build.  TeamCity reported the build as successful, with 3 muted tests.
  3. I fixed the 3 failing tests, ran a build.  All tests passed, no tests shown as muted.  
    So far, everything is as I expected.

  4. I re-created the same 3 failing tests and ran a build.  TeamCity reported the build as successful, with those 3 tests as muted.  


Is the result in step 4 the correct behaviour?  I had expected TeamCity to forget about the previous "muting" on those tests once those tests had passed, and fail the build.

Also, the muted tests in Step 4 do not have the "muted" icon.

I am running TeamCity 8.0.3 Enterprise.  Screenshot of Step 4 results attached.

TCMutedTestsUnexpected.png

5 comments

Hello Ray,

I reproduced your scenario in the same TC version and it works as intended - recreated tests (step 4) failed with no mute flag set on. What tests do you use? Do you have some history builds? Can you please also check server and agent logs for some errors?

regards

Zdenek

0

Hi Zdenek,

Thank-you for your response -- it confirms what I thought was the expected behaviour.  I had been wondering if what I was seeing was a mechanism for dealing with transient failures, but no.

The server and agent logs show no errors.  And no history builds involved.

When I re-ran the build this morning - the same content; absolutely no changes - it behaved as it should: it failed with 3 test failures.  Please see the attached screen shot.  The circled build (build 49) is the build referenced in Step 4 in my previous post.

When I came in this morning, I reran the build (no changes to content or configuration), and the result was as it should be: the build failed with 3 failed tests (build 50).

I then experimented over the day as follows, with builds 59, 67, 68 and onward having incorrect results:

build 50: after the build completed, I muted the 3 tests.
build 51 & 52: no changes; pass with 3 muted tests.
build 53: changed the tests to pass; the build passed, no muted tests.
build 54: no changes; the build passed, no muted tests.
build 55: changed the tests to fail; the build failed with 3 failed tests.
build 56: no changes; the build failed with 3 failed tests.
   -- after the build, I muted the 3 tests.
build 57: no changes; pass with 3 muted tests.
build 58: changed the tests to pass; the build passed, no muted tests.
build 59: changed the tests to fail; the build passed with 3 muted tests!!!!
build 60: no changes; the build now fails (but it was a different agent...).
build 61: no changes; reran the build on the same agent as build 59: fail (so not the agent).
build 62: changed the tests to pass; build passes, no muted tests.
build 63: changed the tests to fail; build fails with 3 failed tests.
build 64 & 65: oops - At this point, I realized I forgot to mute the tests on build 61.
   -- at this point, I muted the 3 failed tests on build 63.
build 66: changed the tests to pass; build passes, no muted tests.
build 67: changed the tests to fail; build passes with 3 muted tests!!!!
build 68 & onward: no changes; failed with 2 failed tests but 1 muted!

During the day, I noticed "mute has been automatically removed" entries in the server log.  By the end of the day, there were these 3:

[2013-10-11 12:05:02,938]   INFO -   jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 3 tests in Dev :: RBird
[2013-10-11 14:50:20,754]   INFO -   jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 3 tests in Dev :: RBird
[2013-10-11 19:19:43,116]   INFO -   jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 2 tests in Dev :: RBird

The first message occurs between builds 54 & 55.
The second between builds 59 & 60.
The last after build 67 and during build 68 through 70.

My hypothesis: after a build successfully runs failed tests, there is an approximately 30 minute wait time before TeamCity "clears" the mute for the project.  Except: why did it clear only 2 of the 3 tests in the last go-round?

Now that I have some notion of what is going on, this is just a minor issue that I can work around.

You asked about the tests I was running: JUnit and CUnit.  The build has 5 steps, 2 of which are disabled.  The JUnit and CUnit tests are run using ant in build step 4.  I have a custom ant task that parses the output from the CUnit tests as they occur and sends services messages to TeamCity.  I've rigged the ant script so that it will not fail if there are JUnit or CUnit test failures.  Of the 3 tests that I have been using in the above, one is a JUnit test and the other two are one CUnit test that gets run in two different situations.

Ray



Attachment(s):
BuildConfigStep4.png
BuildConfigBuildSteps.png
teamcity-server - 2013-10-04-thru-10-11.zip
UnexpectedTestMuting.png
0
Hello Ray,

There should not be any significant delays unmuting the tests (not 30 minutes or so).
There were some issues with incorrect automatic unmuting in 8.0.x, so just in case it's better to upgrade to the latest minor release (8.0.4).
Isn't it possible the test were unmuted because of some other build, which you didn't take into account? Also there is a useful hint: when you think the test should be unmuted, and it is not, you can go to the "Muted problems" tab and expand the test. TeamCity should show all the latest test runs, and since TeamCity thinks it's not successful, there should be a failure among them. It is possible that because of a bug last test runs are computed incorrectly (in this case any details will be helpful for us), i.e. the failed test run is wrong. Or you may spot the test run that you didn't take into account before. But in any case it would be clearer why TeamCity holds the mute.

regards

Zdenek
0

Hi Zdenek,
So, I reproduced the problem and checked the Muted Problems tab - see attached pic "AllTestsPassed2Muted".  I included the clock to remind myself that this was the state after a build had been run where all tests passed successfully after having been muted before running the build.  None of the muted tests are "active": clicking on them does not expand them to show more information.  And then this tab becomes empty after 30 minutes or so.

I did check the Muted Problems tab before running the successful build (sorry, forgot to take a pic), and the muted tests appeared there and could be expanded to show their failure details and the build they failed in, which was the build that they did fail in.  No evidence that I could see of any other build configuration being involved.

I will follow your suggestion and upgrade to 8.0.4 this weekend.

Regards,
Ray



Attachment(s):
AllTestsPassed2Muted.png
0

Hi Zdenek,
I upgraded to 8.0.4, and the issue is still there.

To summarize, the problem manifests by following this sequence of steps:
1. run a build with 3 tests that I rigged to fail.  (Build #0.25.l.22241.eval02.84 in attached pic MutedTests-allBuilds.png).
2. after the build, mute the 3 tests, to be unmuted when next successfully run.
3. run a build where the 3 tests are rigged to pass.  (Build #...85)
4. run a build where the 3 tests are rigged to fail.  (Build #...86)  If the tests within the build are run (and fail) before about 30 minutes after the successful build, the failing tests will be muted (see attached MutedTests-3.png).

Notes:
- As before, there was this message in the log, about 30 minutes after the successful build (step 3 above):
INFO -   jetbrains.buildServer.SERVER - Mute has been automatically removed by TeamCity for 3 tests in Dev :: RBird

- I checked the project's Muted Problems tab several times.  Between steps 2 and 3 above, the muted problems tab shows all the details of the failed tests, including the build it failed in (see attached MutedTests-1.png).  After step 3, the failed tests are still shown, but there are no details (see attached MutedTests-2.png).  After step 4, after more the 30 minutes after the successful build, nothing is shown.   I was running no other builds in my project for several days, and there was no indication that there were any other builds involved.

Again, this is just a minor issue for us, readily lived with.

Regards,
Ray



Attachment(s):
MutedTests-allBuilds.png
MutedTests-3.png
MutedTests-2.png
MutedTests-1.png
teamcity-server.zip
0

Please sign in to leave a comment.