Server infinite loop
I have the server stuck in a loop, trying to stop a build on an agent, after poking at it a little I restarted both the build machine and the server, when I startup the server again on the mainpage it just keeps looping on stopping the build.
#1.5.0.3183 Cancelled (Tests passed: 510, ignored: 94) No artifacts No changes started: 16 Sep 08:26 Stopping
Below are my logs,
Thanks,
Orn
Please sign in to leave a comment.
Why do you think the server was in infinite loop? Have you noticed high CPU load or the server was not responsive?
Could you please attach agent logs too?
--
Pavel Sher
Attached are the build agent logs.
What we see on the mainpage after canceling the build is the following, it just seems to be stuck trying to unload the build.
#1.5.0.3183 Cancelled (Tests passed: 510, ignored: 94) No artifacts No changes
started: 16 Sep 08:26
started: 16 Sep 08:26
Stopping
And the green circular icon just keeps spinning, with a tag "Build is running"
And the warning icon gives this tag
"Build agent .114 was disconnected while running a build.
Please check the agent and network connectivity. Either ensure the agent is connected or stop the build.
Last message was received on: 22 Sep 12:31 (16m:35s ago)"
On the server looking at the cpu load tomcat6.exe is taking 90-99% cpu with about 90MB memory.
Thanks,
Orn
Attachment(s):
buildAgent.zip
Please do the following: http://www.jetbrains.net/confluence/display/TCD3/Reporting+Issues#ReportingIssues-hang
and attach here thread dump from the server.
--
Pavel Sher
Here is a dump from AdaptJ
Attachment(s):
tc_dump.txt
It seems the server hangs on loading build log, probably the log file is broken. Try to locate this log on disk and rename the file. Build logs are stored in the .BuildServer/system/messages folder. There is a number of directories with files with names ]]>.msg3. Build id you can discover from the link to your build.
--
Pavel Sher
We solved the problem already as we needed a build for the testers, probably not recommended but we went into the database after stopping the server and wiped the
TEAMCITY.RUNNING
table clean, then we added the build in this table
TEAMCITY.PAUSED_BUILD_TYPES
after that we were able to start the build again from the main page.
Thanks,
Orn
Hello,
The problem will be fixed in the upcoming Calcutta EAP build.
Regards,
KIR
In the meantime you can review our solution that Orn mentions above below.
http://getonthetrail.blogsite.org/Blog/post/2008/09/Teamcity-Continuous-Loop-Stopping-Build.aspx