build timeout reached but "cannot stop"

Hi,

We have a build timeout of 90mins configured for one of our builds, however the agent seems to be having difficulty stopping the build, and within the UI the message "cannot stop" appears next to the build.

From the agent log:
[2015-11-19 10:29:15,122]   INFO - gent.impl.BuildRunTimeoutTimer - The build New-Apps-Linux #3305 {builId=420233} has been running for more than 90 minutes. Terminating...
[2015-11-19 10:29:15,122]   INFO - nt.impl.BuildRunAgentStateImpl - Proceeding to stop build, stop reason: BuildInterruptReason.EXECUTION_TIMEOUT
[2015-11-19 10:29:15,122]   WARN - s.agent.StacktracesAgentPlugin - Failed to get thread dumps on build termination. No build is running.
[2015-11-19 10:29:15,122]   INFO - uildStages.BuildStagesExecutor - Interrupt command sent to class jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage
[2015-11-19 10:29:15,122]   INFO - ildStages.RunnerStagesExecutor - Interrupt command sent to class jetbrains.buildServer.agent.impl.buildStages.runnerStages.start.CallRunnerStage
[2015-11-19 11:12:01,712]   INFO - nt.impl.BuildRunAgentStateImpl - Another attempt to stop build, stop reason: BuildInterruptReason.SERVER_STOP_BUILD, previous stop reason: BuildInterruptReason.EXECUTION_TIMEOUT
[2015-11-19 11:12:01,712]   INFO - uildStages.BuildStagesExecutor - Interrupt command sent to class jetbrains.buildServer.agent.impl.buildStages.startStages.steps.ForEachBuildRunnerStage
[2015-11-19 11:12:01,713]   INFO - ildStages.RunnerStagesExecutor - Interrupt command sent to class jetbrains.buildServer.agent.impl.buildStages.runnerStages.start.CallRunnerStage


Our only option appears to be doing kill -9 on the agent pid.

Please advise/help... I've seen similar posts like this but no root cause appears to have been identified.

Thanks,
Joe
4 comments

Hi Joe,

What TeamCity version is used? If not the latest one, please consider the upgrade. If the issue is reproduced on the latest version, then please attach full teamcity-agent.log coverging the time of the issue (better in debug mode).

0

Hi - we are not in a position to upgrade yet, possibly early in the new year.
Can nothing be investigated or advised in the meantime?

Thanks,
Joe

0

We are running version 9.0.1 by the way...

Thanks,
Joe

0

On which state does the build hang? Please attach full teamcity-agent.log covering the time of the issue.

0

Please sign in to leave a comment.