Cannot stop agent

Hi,

I'm using TeamCity 4.5.5, server is on Linux, and I have agents on
Windows and Linux.

Sometimes builds are too long, and I configured TeamCity to stop too
long builds. It works well on Windows, but on Linux, from time to time,
the build cannot be stop, I cannot reproduce, I don't know why it happens.

I see this :

This build is probably hanging. Last message was received on: 30 nov. 09
14:07 (17m:18s ago)

Started: 30 nov. 09 13:37    Cannot stop


Is it a known problem?  What can I check to know why it's happening ?

Thanks,

Gérald

20 comments
Comment actions Permalink

Hi,

Sometimes builds are too long, and I configured TeamCity to stop too
long builds. It works well on Windows, but on Linux, from time to time,
the build cannot be stop, I cannot reproduce, I don't know why it happens.

>

I see this :

>

This build is probably hanging. Last message was received on: 30 nov. 09
14:07 (17m:18s ago)

>

Started: 30 nov. 09 13:37    Cannot stop

>
>

Is it a known problem?  What can I check to know why it's happening ?


Anyone can help? That's a pretty blocking problem, even if it doesn't
happen often...

Thanks,

Gérald

0
Comment actions Permalink

Please provide logs from agent when this happens again, specifically: teamcity-agent.log

0
Comment actions Permalink

Le 03/12/2009 13:26, Pavel Sher a ecrit :

Please provide logs from agent when this happens again, specifically: teamcity-agent.log


Ok, I'll do that next time.

Gérald

0
Comment actions Permalink

Got the same problem.So I also would like to know how to stop the agen, but also I can't understand why this build was hanging (here are logs of this build):

[22:26:01]: Checking for changes
[22:26:01]: Clearing temporary directory: /opt/gftcc/teamcity3/temp/buildTmp
[22:26:01]: Checkout directory: /opt/gftcc/teamcity3/work/e4c3228c9738c6b
[22:26:01]: Updating sources (16h:07m:45s)
[22:26:01]: [Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[22:26:01]: [Updating sources] Building clean patch for VCS root: Branch-249 (SVN)
[22:28:33]: [Updating sources] Transferring cached clean patch for VCS root: Branch-249 (SVN)
[22:30:39]: [Updating sources] Transferring repository sources: 101.33Mb so far...
[22:32:46]: [Updating sources] Transferring repository sources: 101.69Mb so far...
[22:35:01]: [Updating sources] Transferring repository sources: 101.70Mb so far...
[22:35:40]: [Updating sources] Failed to build patch for build #40, build id: 10651, VCS root: Branch-249 (SVN), due to error: ClientAbortException: java.net.SocketException: Connection reset
[14:33:47]: [Updating sources] Build cancelled



teamcity-agent.log is attached. Build hung yesterday (03.12.2009) approx at 22.35. Agent was stopped with commands stop kill today approx at 14.40.



Attachment(s):
teamcity-agent.log.zip
0
Comment actions Permalink

It looks like agent was not able to establish reliable connection to the server. Sometimes there are "Proxy error" messages. Do you have some kind of a proxy betwen agent and TeamCity server? It would be better to bypass proxy in this case.

0
Comment actions Permalink

Here is the log agent, I had the problem again...

It seems that some processes can not be killed by build agent, it was yesterday afternoon, but when I look now, the processes are not running anymore, but build is still running, no way to stop it, except restarting agent.



Attachment(s):
teamcity-agent.log.zip
0
Comment actions Permalink

The issue is connected with ps call that was not finished to 14 seconds. Was the machine 100% busy?

0
Comment actions Permalink

Le 10/12/2009 15:57, Gérald Fauvelle a ecrit :

Here is the log agent, I had the problem again...

>

It seems that some processes can not be killed by build agent, it was yesterday afternoon, but when I look now, the processes are not running anymore, but build is still running, no way to stop it, except restarting agent.


Did you find something with this log message ?

Gérald

0
Comment actions Permalink

I've noticed that kill command failes to kill some processes create from teamcity.sh files. Does that file
changes the user while building?

0
Comment actions Permalink

Le 29/12/2009 17:41, Eugene Petrenko a ecrit :

I've noticed that kill command failes to kill some processes create from teamcity.sh files. Does that file
changes the user while building?


No, it doesn't change user, it only call other scripts that call make,
and tests scripts that call executables.

All is done with current user.

Gérald

0
Comment actions Permalink

I've had this same problem for months.
I was using 4.5.5.
Upgrading to version 5 did not help.

0
Comment actions Permalink

I have found that if the agent is running on a remote windows box, the cancel works.
If the agent is running on the same box (in my case linux) as the tc server, the cancel also works.

If the agent is running on a remote solaris, the cancel times out and build appears to be hanging.

Are the folks at Jetbrains able to produce this problem?

0
Comment actions Permalink

Le 04/01/2010 22:19, rick marry a ecrit :

I had this same problem for months.  I upgraded to tc 5 and the problem went away.


Ok, good to know. I've downloaded last version, but I didn't installed
it yet. I'll try and let you know.

Gérald

0
Comment actions Permalink

Version 5 did not fix the problem.  I saw that I was able to cancel a build on a windows agent quickly and thought that version 5 was responsible for that.

0
Comment actions Permalink

When I try to cancel, I get this in the agent log:

[2010-01-05 08:59:54,903]   WARN - erver.agent.impl.GuardedAction - runGuarded failed to acquireStoppingLock() for jetbrains.buildServer.agent.impl.BuildRunAction$1$1@2d6b76

this agent is running on a solaris zone.

0
Comment actions Permalink

I am able to cancel an agent running on a remote linux or windows machine.

I am only unable to cancel an agent running on a solaris machine and here is what is being written to the agent log when i try:
[2010-01-05 08:59:54,903]   WARN - erver.agent.impl.GuardedAction - runGuarded failed to acquireStoppingLock() for jetbrains.buildServer.agent.impl.BuildRunAction$1$1@2d6b76

0
Comment actions Permalink

Could you please provide all agent logs?

0
Comment actions Permalink

I let the build run through once to completion, then I ran again and cancelled half way.

Notice that the maven build completed successfully, on tc, the build still says cannot stop and appears to still be running.
It will continue like this until I manually kill the agent, then restart teamcity and then cancel the build from teamcity.



Attachment(s):
logs.zip
0
Comment actions Permalink

Unfortunately logs did not bring anything new. I would suggest to enable debug logging on the agent, reproduce the bug again and submit bug report to our tracker with logs attached. Also please provide more details about this OS. Read more on how to enable debug logging on the agent: http://www.jetbrains.net/confluence/display/TCD5/Reporting+Issues#ReportingIssues-LoggingEvents

0

Please sign in to leave a comment.