Linux: Build Agent exits with code 23 and restarts periodically

Answered

We're running TC for some years now on Windows machines. Currently we're about to setup a Linux (Ubuntu) machine to build for QNX targets.

 

Problem:

The build agent exits periodically with code 23 approx. every 22 seconds and will be restarted afterwards.

Logs attached.

TeamCity Enterprise 9.1.3 (build 37176)

Agent runs on Ubuntu 16.04  Xenial 32bit.

 

Thanks, Markus

 

(error.log snippet)

Too long run time, unexpected exit. Restarting. Run time was : 25.596 sec.
Too long run time, unexpected exit. Restarting. Run time was : 22.345 sec.
Too long run time, unexpected exit. Restarting. Run time was : 21.897 sec.
Too long run time, unexpected exit. Restarting. Run time was : 25.489 sec.
Too long run time, unexpected exit. Restarting. Run time was : 22.464 sec.

 

 

(launcher.log snippet)

[2016-09-09 05:58:54,209] INFO - dServer.agent.run.AgentProcess - Agent process finished
[2016-09-09 05:58:54,209] INFO - ldServer.agent.run.AgentRunner - Agent has exited with code: 23
[2016-09-09 05:58:54,210] ERROR - ldServer.agent.run.AgentRunner - Too long run time, unexpected exit. Restarting. Run time was : 17.39 sec.
[2016-09-09 05:58:54,210] INFO - ldServer.agent.run.AgentRunner - Launcher is restarting agent process.
[2016-09-09 05:58:54,210] INFO - dServer.agent.run.AgentProcess - Fri Sep 09 05:58:54 CEST 2016 ==> Start launch
[2016-09-09 05:58:54,210] INFO - dServer.agent.run.AgentProcess - Launching agent with command: /usr/lib/jvm/java-9-openjdk-i386/bin/java -ea -Xmx384m -Dteamcity_logs=../logs/ -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml -classpath /opt/tcBuildAgent/lib/freemarker.jar:/opt/tcBuildAgent/lib/common-runtime.jar:/opt/tcBuildAgent/lib/util.jar:/opt/tcBuildAgent/lib/jaxen-1.1.1.jar:/opt/tcBuildAgent/lib/jdom.jar:/opt/tcBuildAgent/lib/log4j-1.2.12-json-layout-1.0.9.jar:/opt/tcBuildAgent/lib/server-logging.jar:/opt/tcBuildAgent/lib/log4j-1.2.12.jar:/opt/tcBuildAgent/lib/agent-upgrade.jar:/opt/tcBuildAgent/lib/coverage-agent-common.jar:/opt/tcBuildAgent/lib/launcher.jar:/opt/tcBuildAgent/lib/messages.jar:/opt/tcBuildAgent/lib/agent-installer-ui.jar:/opt/tcBuildAgent/lib/trove4j.jar:/opt/tcBuildAgent/lib/ehcache-1.6.0-patch.jar:/opt/tcBuildAgent/lib/trove-3.0.3.jar:/opt/tcBuildAgent/lib/coverage-report.jar:/opt/tcBuildAgent/lib/patches-impl.jar:/opt/tcBuildAgent/lib/resources_en.jar:/opt/tcBuildAgent/lib/commons-io-1.3.2.jar:/opt/tcBuildAgent/lib/xpp3-1.1.4c.jar:/opt/tcBuildAgent/lib/jdk-searcher.jar:/opt/tcBuildAgent/lib/commons-logging.jar:/opt/tcBuildAgent/lib/commons-collections-3.2.1.jar:/opt/tcBuildAgent/lib/commons-compress-1.9.jar:/opt/tcBuildAgent/lib/xmlrpc-2.0.1.jar:/opt/tcBuildAgent/lib/openapi.jar:/opt/tcBuildAgent/lib/ehcache-1.6.2.jar:/opt/tcBuildAgent/lib/xstream-1.4.1.jar:/opt/tcBuildAgent/lib/idea-settings.jar:/opt/tcBuildAgent/lib/agent-openapi.jar:/opt/tcBuildAgent/lib/serviceMessages.jar:/opt/tcBuildAgent/lib/slf4j-api-1.7.5.jar:/opt/tcBuildAgent/lib/agent-launcher.jar:/opt/tcBuildAgent/lib/commons-codec-1.3.jar:/opt/tcBuildAgent/lib/launcher-api.jar:/opt/tcBuildAgent/lib/patches.jar:/opt/tcBuildAgent/lib/inspections-util.jar:/opt/tcBuildAgent/lib/commons-beanutils-core.jar:/opt/tcBuildAgent/lib/commons-httpclient-3.1.jar:/opt/tcBuildAgent/lib/common-impl.jar:/opt/tcBuildAgent/lib/duplicator-util.jar:/opt/tcBuildAgent/lib/agent-configurator.jar:/opt/tcBuildAgent/lib/xz-1.5.jar:/opt/tcBuildAgent/lib/processesTerminator.jar:/opt/tcBuildAgent/lib/annotations.jar:/opt/tcBuildAgent/lib/agent.jar:/opt/tcBuildAgent/lib/slf4j-log4j12-1.7.5.jar:/opt/tcBuildAgent/lib/xml-rpc-wrapper.jar:/opt/tcBuildAgent/lib/runtime-util.jar:/opt/tcBuildAgent/lib/xercesImpl.jar:/opt/tcBuildAgent/lib/app-wrapper.jar:/opt/tcBuildAgent/lib/spring.jar:/opt/tcBuildAgent/lib/joda-time.jar:/opt/tcBuildAgent/lib/common.jar:/opt/tcBuildAgent/lib/buildAgent-updates-applying.jar jetbrains.buildServer.agent.AgentMain -file ../conf/buildAgent.properties -launcher.version 37176
[2016-09-09 05:59:22,749] INFO - dServer.agent.run.AgentProcess - Agent process finished
[2016-09-09 05:59:22,749] INFO - ldServer.agent.run.AgentRunner - Agent has exited with code: 23

 

(teamcity-agent.log tail)

[2016-09-09 12:57:53,309] INFO - jetbrains.buildServer.SERVER - Starting TeamCity agent
[2016-09-09 12:57:53,309] INFO - s.buildServer.agent.AgentMain2 -

 

 


[2016-09-09 12:57:53,309] INFO - s.buildServer.agent.AgentMain2 - ===========================================================
[2016-09-09 12:57:53,388] INFO - s.buildServer.agent.AgentMain2 - TeamCity Build Agent 9.1.3 (build 37176), OS: Linux, User: root, JRE: 9-internal, OpenJDK Server VM (32 bit) (9-internal+0-2016-04-14-195526.buildd.src, mixed mode), OpenJDK Runtime Environment (9-internal+0-2016-04-14-195526.buildd.src), Oracle Corporation, JVM parameters: -ea -Xmx384m -Dteamcity_logs=../logs/ -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml
[2016-09-09 12:57:53,388] INFO - s.buildServer.agent.AgentMain2 - Starting...
[2016-09-09 12:57:53,504] INFO - buildServer.agent.AgentMain2$2 - Refreshing jetbrains.buildServer.agent.AgentMain2$2@feca44: startup date [Fri Sep 09 12:57:53 CEST 2016]; root of context hierarchy
[2016-09-09 12:57:54,923] INFO - jetbrains.buildServer.AGENT - Calculating CPU benchmark index...
[2016-09-09 12:58:02,267] INFO - jetbrains.buildServer.AGENT - CPU benchmark index is set to 445
[2016-09-09 12:58:02,592] INFO - dAgentConfigurationInitializer - Loading build agent configuration from ../conf/buildAgent.properties
[2016-09-09 12:58:02,651] INFO - s.buildServer.agent.AgentMain2 - Working dir: /opt/tcBuildAgent/work
[2016-09-09 12:58:02,651] INFO - s.buildServer.agent.AgentMain2 - Temp dir: /opt/tcBuildAgent/temp
[2016-09-09 12:58:02,677] INFO - rver.plugins.PluginManagerImpl - ===========================================================
[2016-09-09 12:58:02,677] INFO - rver.plugins.PluginManagerImpl - Plugins initialization started...
[2016-09-09 12:58:02,679] INFO - rver.plugins.PluginManagerImpl - Scanning plugins folders
[2016-09-09 12:58:02,686] INFO - .plugins.files.JarSearcherBase - Scanning plugin folder: /opt/tcBuildAgent/plugins
[2016-09-09 12:58:02,813] INFO - .plugins.files.JarSearcherBase - Scanning plugin folder: /opt/tcBuildAgent/tools
[2016-09-09 12:58:02,866] INFO - rver.plugins.PluginManagerImpl - Found 56 plugins: [agentSystemInfo, amazonEC2, ant, ant-net-tasks, antPlugin, assembly-info-patcher, clearcase-agent, commandLineRunner, coveragePlugin, crashDetector, cvsAgent, dotCover, dotnet-tools, dotnetPlugin, dotNetRunners, duplicatePlugin, environment-fetcher, file-content-replacer, fxcop, gant, gantPlugin, gradle-runner, idea, idea-runner, inspectionPlugin, ipr-runner, jacoco, java-dowser, jetbrains.git, jps, jps-old, junitPlugin, jvm-update, maven, maven3, maven3_1, maven3_2, mavenPlugin, mercurial, meta-runner, nuget-agent, perfmon, perforce-agent, powershell-agent, rake-runner, remoteAccess, sbt-runner-agent, ssh-manager, stacktracesPlugin, svnAgent, swabra, testNGPlugin, tfs-agent, visualstudiotest, xcode-runner, xml-report-plugin]
[2016-09-09 12:58:02,868] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/tools/maven3/bin/mvn
[2016-09-09 12:58:02,868] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/ant
[2016-09-09 12:58:02,868] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/antRun
[2016-09-09 12:58:02,868] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/ant.bat
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/ant.cmd
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/antenv.cmd
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/envset.cmd
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/lcp.bat
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/runrc.cmd
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/antRun.pl
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/complete-ant-cmd.pl
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/runant.pl
[2016-09-09 12:58:02,869] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/plugins/ant/bin/runant.py
[2016-09-09 12:58:02,870] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/tools/maven/bin/mvn
[2016-09-09 12:58:02,870] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/tools/maven3_2/bin/mvn
[2016-09-09 12:58:02,870] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/tools/maven3_1/bin/mvn
[2016-09-09 12:58:02,871] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/tools/gant/bin/gant
[2016-09-09 12:58:02,871] INFO - les.ExecutableAttributesSetter - Setting executable bit for: /opt/tcBuildAgent/tools/gant/bin/startGroovy
[2016-09-09 12:58:02,871] INFO - rver.plugins.PluginsCollection - Load shared classloader for 28 plugins [agentSystemInfo, amazonEC2, ant, assembly-info-patcher, clearcase-agent, commandLineRunner, crashDetector, cvsAgent, dotnet-tools, dotnetPlugin, dotNetRunners, duplicatePlugin, environment-fetcher, file-content-replacer, fxcop, gradle-runner, inspectionPlugin, java-dowser, jetbrains.git, mercurial, perfmon, powershell-agent, remoteAccess, ssh-manager, stacktracesPlugin, swabra, tfs-agent, xml-report-plugin]
[2016-09-09 12:58:02,899] INFO - rver.plugins.PluginsCollection - Load standalone classloaders for 17 plugins [antPlugin, coveragePlugin, gantPlugin, idea-runner, ipr-runner, junitPlugin, jvm-update, mavenPlugin, meta-runner, nuget-agent, perforce-agent, rake-runner, sbt-runner-agent, svnAgent, testNGPlugin, visualstudiotest, xcode-runner]

 

1 comment

The issue was resolved by switching to JRE 8. 

At the moment JRE 9 is not supported, the related ticket is: https://youtrack.jetbrains.com/issue/TW-47064.

0

Please sign in to leave a comment.