Problem running build -- java.lang.OutOfMemoryError: Java heap space

Hi Scott,

I suppose the problem with OOM can be on the server side. Could you please take a look at
TeamCity/logs/teamcity.log - does it possibly contain OOM errors? If so, I'd suggest increasing
Xmx for the TeamCity server.

Currently, there is no way to specify Xmx for agent itself and there is a separate issue for that:
http://www.jetbrains.net/jira/browse/TW-1473

BTW, that's cool you're up and running with 11 agents :)

Regards,
KIR

scoheb wrote:

Hi,

I am running TeamCity 1.2 (1784) and one of my agents seems to be dying halfway thru one build:

The agent log file contains this:

+ INFO - r.agent.impl.AgentLogProxyImpl - jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://cgnsps01.cae.ca:8080/teamcity/RPC2 buildServer.log: java.lang.OutOfMemoryError: Java heap space
jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://cgnsps01.cae.ca:8080/teamcity/RPC2 buildServer.log: java.lang.OutOfMemoryError: Java heap space
at jetbrains.buildServer.xmlrpc.XmlRpcTargetImpl.call(XmlRpcTargetImpl.java:69)
at jetbrains.buildServer.agent.impl.XmlRpcAgentLogListener.log(XmlRpcAgentLogListener.java:37)
at jetbrains.buildServer.agent.impl.AgentLogProxyImpl.sendLog(AgentLogProxyImpl.java:142)
at jetbrains.buildServer.agent.impl.AgentLogProxyImpl.access$500(AgentLogProxyImpl.java:21)
at jetbrains.buildServer.agent.impl.AgentLogProxyImpl$5.run(AgentLogProxyImpl.java:172)
at jetbrains.buildServer.agent.impl.AgentLogProxyImpl.sendActionRequest(AgentLogProxyImpl.java:66)
at jetbrains.buildServer.agent.impl.AgentLogProxyImpl.access$100(AgentLogProxyImpl.java:21)
at jetbrains.buildServer.agent.impl.AgentLogProxyImpl$2.run(AgentLogProxyImpl.java:93)
at jetbrains.buildServer.agent.impl.AgentLogProxyImpl$LaterExecutorTask.run(AgentLogProxyImpl.java:226)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space+

This build configuration uses the ANT runner and it is configured with these params: -Xmx512m -XX:MaxPermSize=256m

The server has about 11 agents, most of which do not exhibit this behavior.

Could this be a problem with the agent itself? I thought it was due to the agent running out of memory and I tried creating a agentd.exe.vmoptions file in /bin containing:

+-Xmx512m
-J-Xmx512m+

But it seemed to have no effect.

Thanks,

Scott

--
Kirill Maximov
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

Kirill Maximov (JetBrains) wrote:

BTW, that's cool you're up and running with 11 agents :)

Any support yet for sharing a build agent install on a network drive or
do I still have to do individual installs on the machine concerned?
R

I'm trying to run the same build but with the agent started with agent.bat. It seems to allow one to specify the Xmx param.

Have you considered controlling the agent using the Java Service Wrapper?

Thanks,

Scott

I have re-run the build on the agent in question using agent.bat and increasing -Xmx.

The build completes successfully.

This was not a server issue but an agent memory issue.

Any chance http://www.jetbrains.net/jira/browse/TW-1473 can be backported to 1.2 :>

Thanks,

Scott

Robert,

For a build agent, you can specify buildAgent.properties file using -f key in command line.
So you can possibly have different properties files for different agents and share the same build agent installation (though we didn't test such a scenario).

There could be troubles with agent auto-upgrade, because in such case several computers will try to upgrade the same build agent installation.

Each build agent must have separate work and temp directories and they already can be configured via buildAgent.properties file. The same applies to build agent name.

What is your vision for shared agent code? Why would you need it given that build agents can autoupgrade?

Regards,
KIR

Hi Kirill,

Kirill Maximov (JetBrains) wrote:

For a build agent, you can specify buildAgent.properties file using -f key in command line.

OK that's nearly what I need, I might give it a try with a couple of
machines and see how it works (watching out for upgrades). The ideal for
me is to be able to share config files and everything (e.g. with the
agent name being the machine name if not specified).

What is your vision for shared agent code? Why would you need it given that build agents can autoupgrade?

We already have a grid of machines set up and all the software that runs
on them runs off the shared network drive. When administrating the
machines it is much easier to not have per-machine configurations.
Auto-upgrade will ease part of the pain, but not all of it.

Thanks,
R

Hi,

I'm afraid, we have no plans to release 1.2.+, the whole team works intensively on Agra.

Kind regards,
KIR

Kirill Maximov (JetBrains) wrote:

Each build agent must have separate work and temp directories

... and log directories :(

I managed to get things up and running with a small wrapper script that
creates an individual config file per-machine and calls agent.sh with a
bit of hacking http://www.jetbrains.net/jira/browse/TW-1635

But the location of the teamcity.log file seems to be specified in the
log4j config file and the location of the log4j config file seems to be
hard-coded in the agent, is that right? Is there a way to specify this
on the command line?

Thanks,
R

If anyone else is interested, I got this working by hacking the agent.sh
script as shown in TW-1635 and putting this wrapper script below in the
same directory as the (modified) agent.sh script.

#!/bin/sh
cd `dirname $0`
HOSTNAME=`hostname -s`
CONFIG_DIR=../conf
PROPERTY_FILE=$CONFIG_DIR/buildAgent.properties.$HOSTNAME

if [[ ! -r $PROPERTY_FILE ]]; then
sed -e s/HOSTNAME/$HOSTNAME/ <
$CONFIG_DIR/buildAgent.properties.template > $PROPERTY_FILE
fi
export CONFIG_FILE=$PROPERTY_FILE
export LOG_DIR=~/buildAgent/logs.$HOSTNAME
agent.sh $*


You will also need a file called buildAgent.properties.template in the
conf directory that looks something like this
name=HOSTNAME
ownPort=9090
serverUrl=http://buildserver:8111/
workDir=/tmp/work

HTH,
R

Kirill Maximov (JetBrains) wrote:
But the location of the teamcity.log file seems to be
specified in the
log4j config file and the location of the log4j
config file seems to be
hard-coded in the agent, is that right? Is there a

Wrong. We have teamcity_log_config property to specify name and location of log4.xml configuration file. You may also leave this file intact and specify property teamcity_logs with location of log directory (see content of teamcity-log4j.xml on how we use teamcity_logs property).

way to specify this
on the command line?

Thanks,
R

>

No, we didn't look at Java Service Wrapper, but we have an issue for specifying Xmx for build agent service.

Regards,
KIR

Kirill Maximov (JetBrains) wrote:

You may also leave this file intact and specify property teamcity_logs with location of log directory (see content of teamcity-log4j.xml on how we use teamcity_logs property).

This is what I ended up playing with, thanks.
R

No, we didn't look at Java Service Wrapper, but we
have an issue for specifying Xmx for build agent
service.

Regards,
KIR

I've setup my build agent to use the Java Service Wrapper. Would you like me to forward you the config files and setup?

Thanks,

Scott

I've setup my build agent to use the Java Service
Wrapper. Would you like me to forward you the config
files and setup?

I'd appreciate if you do. Please attach it to http://www.jetbrains.net/jira/browse/TW-1473 , we'll consider move to the wrapper instead of existing agentd service.

Thanks for the help,
KIR

As promised, I have attached my setup to http://www.jetbrains.net/jira/browse/TW-1473 in order to use the Java Service Wrapper with the TeamCity Build Agent. I've only include the Windows binaries but the config file supports all platforms.

To install service -- wrapper.exe -i ../conf/wrapper.conf
To start service -- wrapper.exe -t ../conf/wrapper.conf
To stop service -- wrapper.exe -p ../conf/wrapper.conf
To remove service -- wrapper.exe -r ../conf/wrapper.conf

Note that the Linux/UNIX packages available at http://wrapper.tanukisoftware.org/ contain the init.d startup scripts.

Please don't hesitate to contact me if you have any questions.

Scott