Cannot stop agent
Hi,
I'm using TeamCity 4.5.5, server is on Linux, and I have agents on
Windows and Linux.
Sometimes builds are too long, and I configured TeamCity to stop too
long builds. It works well on Windows, but on Linux, from time to time,
the build cannot be stop, I cannot reproduce, I don't know why it happens.
I see this :
This build is probably hanging. Last message was received on: 30 nov. 09
14:07 (17m:18s ago)
Started: 30 nov. 09 13:37 Cannot stop
Is it a known problem? What can I check to know why it's happening ?
Thanks,
Gérald
Please sign in to leave a comment.
Hi,
>
>
>
>
>
Anyone can help? That's a pretty blocking problem, even if it doesn't
happen often...
Thanks,
Gérald
Please provide logs from agent when this happens again, specifically: teamcity-agent.log
Le 03/12/2009 13:26, Pavel Sher a ecrit :
Ok, I'll do that next time.
Gérald
Got the same problem.So I also would like to know how to stop the agen, but also I can't understand why this build was hanging (here are logs of this build):
[22:26:01]: Clearing temporary directory: /opt/gftcc/teamcity3/temp/buildTmp
[22:26:01]: Checkout directory: /opt/gftcc/teamcity3/work/e4c3228c9738c6b
[22:26:01]: Updating sources (16h:07m:45s)
[22:26:01]: [Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[22:26:01]: [Updating sources] Building clean patch for VCS root: Branch-249 (SVN)
[22:28:33]: [Updating sources] Transferring cached clean patch for VCS root: Branch-249 (SVN)
[22:30:39]: [Updating sources] Transferring repository sources: 101.33Mb so far...
[22:32:46]: [Updating sources] Transferring repository sources: 101.69Mb so far...
[22:35:01]: [Updating sources] Transferring repository sources: 101.70Mb so far...
[22:35:40]: [Updating sources] Failed to build patch for build #40, build id: 10651, VCS root: Branch-249 (SVN), due to error: ClientAbortException: java.net.SocketException: Connection reset
[14:33:47]: [Updating sources] Build cancelled
teamcity-agent.log is attached. Build hung yesterday (03.12.2009) approx at 22.35. Agent was stopped with commands stop kill today approx at 14.40.
Attachment(s):
teamcity-agent.log.zip
It looks like agent was not able to establish reliable connection to the server. Sometimes there are "Proxy error" messages. Do you have some kind of a proxy betwen agent and TeamCity server? It would be better to bypass proxy in this case.
Here is the log agent, I had the problem again...
It seems that some processes can not be killed by build agent, it was yesterday afternoon, but when I look now, the processes are not running anymore, but build is still running, no way to stop it, except restarting agent.
Attachment(s):
teamcity-agent.log.zip
The issue is connected with ps call that was not finished to 14 seconds. Was the machine 100% busy?
Le 10/12/2009 15:57, Gérald Fauvelle a ecrit :
>
Did you find something with this log message ?
Gérald
I've noticed that kill command failes to kill some processes create from teamcity.sh files. Does that file
changes the user while building?
Le 29/12/2009 17:41, Eugene Petrenko a ecrit :
No, it doesn't change user, it only call other scripts that call make,
and tests scripts that call executables.
All is done with current user.
Gérald
I've had this same problem for months.
I was using 4.5.5.
Upgrading to version 5 did not help.
I have found that if the agent is running on a remote windows box, the cancel works.
If the agent is running on the same box (in my case linux) as the tc server, the cancel also works.
If the agent is running on a remote solaris, the cancel times out and build appears to be hanging.
Are the folks at Jetbrains able to produce this problem?
Le 04/01/2010 22:19, rick marry a ecrit :
Ok, good to know. I've downloaded last version, but I didn't installed
it yet. I'll try and let you know.
Gérald
Version 5 did not fix the problem. I saw that I was able to cancel a build on a windows agent quickly and thought that version 5 was responsible for that.
When I try to cancel, I get this in the agent log:
[2010-01-05 08:59:54,903] WARN - erver.agent.impl.GuardedAction - runGuarded failed to acquireStoppingLock() for jetbrains.buildServer.agent.impl.BuildRunAction$1$1@2d6b76
this agent is running on a solaris zone.
I am able to cancel an agent running on a remote linux or windows machine.
I am only unable to cancel an agent running on a solaris machine and here is what is being written to the agent log when i try:
[2010-01-05 08:59:54,903] WARN - erver.agent.impl.GuardedAction - runGuarded failed to acquireStoppingLock() for jetbrains.buildServer.agent.impl.BuildRunAction$1$1@2d6b76
Could you please provide all agent logs?
I let the build run through once to completion, then I ran again and cancelled half way.
Notice that the maven build completed successfully, on tc, the build still says cannot stop and appears to still be running.
It will continue like this until I manually kill the agent, then restart teamcity and then cancel the build from teamcity.
Attachment(s):
logs.zip
Unfortunately logs did not bring anything new. I would suggest to enable debug logging on the agent, reproduce the bug again and submit bug report to our tracker with logs attached. Also please provide more details about this OS. Read more on how to enable debug logging on the agent: http://www.jetbrains.net/confluence/display/TCD5/Reporting+Issues#ReportingIssues-LoggingEvents
cranked logging and entered bug report
http://youtrack.jetbrains.net/issue/TW-10698