Flaky test result
Dear Jetbrains team
On our on premise TeamCity (TeamCity Enterprise 2024.03.1 (build 156270)), we observe that some tests are failing when the are executed using the scheduled task. If we trigger the same test manually, all tests pass.
It does not matter when the scheduled tests start, usually at 01:30 in the morning, but our developer found out that regardless when the scheduled run starts, 3 tests fail. If we could trigger the test at the same time, it would pass.
Did someone else have the same issue? What could be the cause for it?
Thanks a lot for any hint
Michael
Please sign in to leave a comment.
Is this build a part of the build chain, or does it run independently? Please also share the screenshots of the schedule trigger settings and manual run settings (if you use run custom build functionality).
Best regards,
Anton
Hi Anton
These tests run independently.
The settings are as follows:
Build branch filter is:
+:<default>
+:release/22.1.x
+:release/22.1.1
+:release/23.1.x
+:release/23.1.1
+:release/24.1.x
+:release/24.1.1
+:release/25.1.0
+:release/25.1.1
+:release/26.1.0
OK, thank you, I don't see anything that could cause this behavior yet. Could you upload the build logs for the successful and failed runs? You can upload them using the https://uploads.jetbrains.com/ and share the upload ID.
Best regards,
Anton
Hi Anton
Here you go
2024_05_16_6ZiHC3xiMmRChzatXwJ174
From the logs, I can see that both builds are building the same revision and branch. However, they are run on different build agents. Successful on zz03wwBuild03.2 and failed on zz03wwBuild01.1. Could you try running the manual build on the zz03wwBuild01.1 build agent and check the results? You can select the build agent from the custom build dialog.
Please also share the screenshot of Build Configuration > Version Control Settings > Checkout settings.
Best regards,
Anton
Thank you. It may be related to the agent. Did you try running the manual build on the zz03wwBuild01.1 build agent? Please let me know the results.
Best regards,
Anton
Triggering it manually works most of the time:
The idea of the test is to build the same branch and the same revision on the same agent but using the scheduler and a manual build.
In the logs you provided, the builds run on the same branch and revision but on different agents. On the screenshot, I can see one successful build on the same agent that failed in the logs, but it has a different branch.
Running the same build under the same conditions but starting it differently should help find the cause of the issue. Please wait for the next failed scheduled build, then run it on the same agent manually and see if it fails.
Also, I can see that on the screenshot, there are successful builds that seemingly run on the schedule (at 1 AM). You mentioned earlier that scheduled builds are failing. Could you provide more details on this? Is this issue not actual anymore?
Best regards,
Anton