Issue with starting agent and exit codes

Hi Team,

We have an issue with cronned starting/stoping of TeamCity Server and several agents on it. We have 1 build agent on server's local site and 2 agents on dedicated servers in different timezone. First of all I need to say that it's not a scheduling or timezone issue. I checked this carefully several times. Every weekend we have build server hardware rebooted so Teamcity Server is scheduled to shutdown and start automatically around that time. It works fine. But after starting server can see only local agent because remote dedicated agents are down. As a result of some investigation in cron, server and agent logs I can say that it looks like remote agents have been started correctly but then for some reason killed themselves!

Here is some piece of agent logs:
[2012-09-09 10:01:54,921]   INFO - ldServer.agent.run.AgentRunner - Agent process finished

[2012-09-09 10:01:54,955]   INFO - ldServer.agent.run.AgentRunner - Agent has exited with code: 30

[2012-09-09 10:01:54,974]   INFO - ldServer.agent.run.AgentRunner - Launcher is exiting

[2012-09-10 12:53:43,545]  DEBUG - buildServer.agent.LauncherUtil - Deleting /somepath/teamcity/buildAgent/update

[2012-09-10 12:53:43,550]  DEBUG - buildServer.agent.LauncherUtil - Deleting /somepath/teamcity/buildAgent/backup

[2012-09-10 12:53:43,570]   INFO - ldServer.agent.run.AgentRunner - Mon Sep 10 12:53:43 BST 2012 ==> Start launch

[2012-09-10 12:53:43,570]  DEBUG - .buildServer.agent.JavaChooser - Current JRE version is 1.6

[2012-09-10 12:53:43,571]   INFO - ldServer.agent.run.AgentRunner - Launching agent with command:  /sbcimp/run/tp/sun/jdk/v1.6.0_07/jre/bin/java -ea -Xmx384m -Dteamcity_logs=../logs/ -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml -classpath /somepath/teamcity/buildAgent/lib/launcher.jar:/somepath/teamcity/buildAgent/lib/duplicator-util.jar:/somepath/teamcity/buildAgent/lib/agentInstaller-UI.jar:/somepath/teamcity/buildAgent/lib/coverage-report.jar:/somepath/teamcity/buildAgent/lib/commons-logging.jar:/somepath/teamcity/buildAgent/lib/spring.jar:/somepath/teamcity/buildAgent/lib/ehcache-1.6.0-patch.jar:/somepath/teamcity/buildAgent/lib/ivy-2.1.0.jar:/somepath/teamcity/buildAgent/lib/slf4j-simple-1.5.11.jar:/somepath/teamcity/buildAgent/lib/inspections-util.jar:/somepath/teamcity/buildAgent/lib/commons-compress-1.1-SNAPSHOT.jar:/somepath/teamcity/buildAgent/lib/utils.jar:/sbclocal/data/Poker/teamcity/buildAgent/lib/idea-settings.jar:/somepath/teamcity/buildAgent/lib/commons-beanutils-core.jar:/somepath/teamcity/buildAgent/lib/commons-io-1.3.2.jar:/sbclocal/data/Poker/teamcity/buildAgent/lib/coverage-agent-common.jar:/somepath/teamcity/buildAgent/lib/annotations.jar:/somepath/teamcity/buildAgent/lib/server-logging.jar:/sbclocal/data/Poker/teamcity/buildAgent/lib/resources_en.jar:/somepath/teamcity/buildAgent/lib/agent-openapi.jar:/somepath/teamcity/buildAgent/lib/freemarker.jar:/somepath/teamcity/buildAgent/lib/trove4j.jar:/somepath/teamcity/buildAgent/lib/xstream-1.3.1.jar:/somepath/teamcity/buildAgent/lib/patches-impl.jar:/somepath/teamcity/buildAgent/lib/xercesImpl.jar:/sbclocal/data/Poker/teamcity/buildAgent/lib/agent.jar:/somepath/teamcity/buildAgent/lib/slf4j-api-1.5.11.jar:/somepath/teamcity/buildAgent/lib/common-impl.jar:/somepath/teamcity/buildAgent/lib/joda-time-1.5.2.jar:/somepath/teamcity/buildAgent/lib/processesTerminator.jar:/somepath/teamcity/buildAgent/lib/messages.jar:/somepath/teamcity/buildAgent/lib/jdk-searcher.jar:/somepath/teamcity/buildAgent/lib/serviceMessages.jar:/somepath/teamcity/buildAgent/lib/common.jar:/somepath/teamcity/buildAgent/lib/xpp3_min-1.1.4c.jar:/somepath/teamcity/buildAgent/lib/jdom.jar:/somepath/teamcity/buildAgent/lib/commons-codec-1.3.jar:/somepath/teamcity/buildAgent/lib/openapi.jar:/somepath/teamcity/buildAgent/lib/commons-collections-3.2.1.jar:/somepath/teamcity/buildAgent/lib/xml-rpc-wrapper.jar:/somepath/teamcity/buildAgent/lib/patches.jar:/somepath/teamcity/buildAgent/lib/agent-launcher.jar:/somepath/teamcity/buildAgent/lib/commons-httpclient-3.1.jar:/somepath/teamcity/buildAgent/lib/util.jar:/somepath/teamcity/buildAgent/lib/log4j-1.2.12.jar:/somepath/teamcity/buildAgent/lib/common-runtime.jar:/somepath/teamcity/buildAgent/lib/jakarta-oro-2.0.8.jar:/somepath/teamcity/buildAgent/lib/ehcache-1.6.0.jar:/sbclocal/data/Poker/teamcity/buildAgent/lib/runtime-util.jar:/somepath/teamcity/buildAgent/lib/xmlrpc-2.0.1.jar:/somepath/teamcity/buildAgent/lib/jaxen-1.1.1.jar:/somepath/teamcity/buildAgent/lib/agentInstaller.jar jetbrains.buildServer.agent.AgentMain -file ../conf/buildAgent.properties -launcher.version 17834

[2012-09-16 10:01:42,014]   INFO - ldServer.agent.run.AgentRunner - Agent process finished

[2012-09-16 10:01:42,051]   INFO - ldServer.agent.run.AgentRunner - Agent has exited with code: 30

[2012-09-16 10:01:42,062]   INFO - ldServer.agent.run.AgentRunner - Launcher is exiting

As you can see around 10 AM build agent dies every week!
I started it manually at 2012-09-10 12:53:43.

Here is corn tasks:
# Weekly shutdown of TeamCity
00 01 * * 0    export JAVA_HOME=/somepath/jdk/v1.6.0_07 ; /somepath/TeamCity/bin/runAll.sh stop &> /tmp/cronTeamCityStop
# Weekly starting of TeamCity
00 17 * * 0    export JAVA_HOME=/somepath/jdk/v1.6.0_07 ; export TEAMCITY_DATA_PATH=/somepath/TeamCity/BuildServer ; ps aux | grep TeamCity | grep -v grep | grep -v Agent || /somepath/TeamCity/bin/runAll.sh start &> /tmp/cronTeamCityStart

And cron logs for starting and stopping you can find in atachments.

I have a feeling that it dies because it was running actually at the moment because was not stopped somehow. You can find details in cronStop logs. I haven't found any clue of connection to remote build agents there.

First of all the questions:
Does server side "runAll -start" and "runAll -stop" affectes remote build agents?
How can I force server to stop remote build agent correctly?
And some other ones:
What does "Agent has exited with code: 30" means and how can we fix this issue?

Any help would be appreciated!



Attachment(s):
cronStop.txt.zip
cronStart.txt.zip
1 comment
Comment actions Permalink

Any update on this please?

0

Please sign in to leave a comment.