[1615] Lots of OutOfMemory Errors

I'm seeing lots of OutOfMemory errors that I was not previously seeing. Both Heap Space and PermGen memory. I know our build is very intensive, but I do bump up the memory usage in the individual build configurations (-Xmx1536m -XX:PermSize=32M -XX:MaxPermSize=512M) and this seems to have been fine. With this latest (and perhaps 1611...don't recall exactly when it started and they came out very close to each other), I've been seeing lots of memory errors.

It's hard to tell where they are coming from. I've tried beefing up the memory available to the agents and server and will see if that works.

--Tim

7 comments
Comment actions Permalink

Hello Tim,

Have you tried building your builds without TeamCity? Is there a chance
that your build became really memory-hungry because of some reason?

Regards,
KIR


Tim McNerney wrote:

I'm seeing lots of OutOfMemory errors that I was not previously seeing. Both Heap Space and PermGen memory. I know our build is very intensive, but I do bump up the memory usage in the individual build configurations (-Xmx1536m -XX:PermSize=32M -XX:MaxPermSize=512M) and this seems to have been fine. With this latest (and perhaps 1611...don't recall exactly when it started and they came out very close to each other), I've been seeing lots of memory errors.

It's hard to tell where they are coming from. I've tried beefing up the memory available to the agents and server and will see if that works.

--Tim



--
Kirill Maximov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

Our builds are memory hungry and we use the jvm args listed.

Further, this wasn't happening with earlier builds of TC and now is.

It is certainly possible that the fault is mostly our build and is simply aggravated by something in the TC environ. I'll keep looking at things.

--Tim

0
Comment actions Permalink

Here are some examples. If you can indicate where these are happening, that would be great.

From TeamCity/logs (I've tried to filter out repeats):

catalina.2006-09-25.log:java.lang.OutOfMemoryError: Java heap space
catalina.2006-09-25.log:SEVERE: Caught exception (java.lang.OutOfMemoryError: Java heap space) executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@3a5f299d, terminating thread
catalina.out:[02:49:56,710] ERROR - verSide.impl.BuildServerRunner - java.lang.OutOfMemoryError: Java heap space
catalina.out:Caused by: java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "Agent Distribution Monitor" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Jabber config" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Projects configuration" Exception in thread "ChangeObserver: Server config" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Projects configuration" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Win32 config" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "DBMaintenanceScheduler" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "Ping" java.lang.OutOfMemoryError: Java heap space
catalina.out:java.lang.OutOfMemoryError: Java heap space
catalina.out:jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://192.168.192.158:9090/RPC2 buildAgent.ping: java.lang.OutOfMemoryError: Java heap space
catalina.out:SEVERE: Caught exception (java.lang.OutOfMemoryError: Java heap space) executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@3a5f299d, terminating thread
teamcity.log:[02:49:56,710] ERROR - verSide.impl.BuildServerRunner - java.lang.OutOfMemoryError: Java heap space
teamcity.log:Caused by: java.lang.OutOfMemoryError: Java heap space
teamcity.log:java.lang.OutOfMemoryError: Java heap space
teamcity.log:jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://192.168.192.158:9090/RPC2 buildAgent.ping: java.lang.OutOfMemoryError: Java heap space
teamcity.log:jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://192.168.192.159:9090/RPC2 buildAgent.ping: java.lang.OutOfMemoryError: Java heap space
teamcity-vcs.log.1:java.lang.OutOfMemoryError: Java heap space
teamcity-vcs.log:java.lang.OutOfMemoryError: Java heap space

Are these being passed on from the agents?

There were others, some of which I don't see anymore after boosting up values in a number of places.

I will quit all the servers and agents and start from scratch to see if I can get better information. I'll also try and do thread dumps when I see errors.

--Tim

0
Comment actions Permalink

Hello Tim,

Sorry for the delay with the answer, we've had troubles between forum <-> news server
synchronization.

It looks like OutOfMemory error occure on the server side.
To debug them, I'd reccomend the following:
- Setup JDK 1.5.0_08 on the server computer
- add options -XX:+HeapDumpOnOutOfMemoryError -Xmx500m to the JAVA_OPTS env. variable parameters
in TeamCity/bin/catalina.sh

With such options java will generate hprof file with memory information at the moment of OOME.
If you upload this file to our ftp site ftp://ftp.intellij.net/.uploads we'll be able to
research the problem in detail.

Please don't use Xmx > 500m, because I'm afraid we won't be able to open corresponding memory
dump file.

Kind regards,
KIR

Tim McNerney wrote:

Here are some examples. If you can indicate where these are happening, that would be great.

From TeamCity/logs (I've tried to filter out repeats):

catalina.2006-09-25.log:java.lang.OutOfMemoryError: Java heap space
catalina.2006-09-25.log:SEVERE: Caught exception (java.lang.OutOfMemoryError: Java heap space) executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@3a5f299d, terminating thread
catalina.out:[02:49:56,710] ERROR - verSide.impl.BuildServerRunner - java.lang.OutOfMemoryError: Java heap space
catalina.out:Caused by: java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "Agent Distribution Monitor" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Jabber config" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Projects configuration" Exception in thread "ChangeObserver: Server config" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Projects configuration" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "ChangeObserver: Win32 config" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "DBMaintenanceScheduler" java.lang.OutOfMemoryError: Java heap space
catalina.out:Exception in thread "Ping" java.lang.OutOfMemoryError: Java heap space
catalina.out:java.lang.OutOfMemoryError: Java heap space
catalina.out:jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://192.168.192.158:9090/RPC2 buildAgent.ping: java.lang.OutOfMemoryError: Java heap space
catalina.out:SEVERE: Caught exception (java.lang.OutOfMemoryError: Java heap space) executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@3a5f299d, terminating thread
teamcity.log:[02:49:56,710] ERROR - verSide.impl.BuildServerRunner - java.lang.OutOfMemoryError: Java heap space
teamcity.log:Caused by: java.lang.OutOfMemoryError: Java heap space
teamcity.log:java.lang.OutOfMemoryError: Java heap space
teamcity.log:jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://192.168.192.158:9090/RPC2 buildAgent.ping: java.lang.OutOfMemoryError: Java heap space
teamcity.log:jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://192.168.192.159:9090/RPC2 buildAgent.ping: java.lang.OutOfMemoryError: Java heap space
teamcity-vcs.log.1:java.lang.OutOfMemoryError: Java heap space
teamcity-vcs.log:java.lang.OutOfMemoryError: Java heap space

Are these being passed on from the agents?

There were others, some of which I don't see anymore after boosting up values in a number of places.

I will quit all the servers and agents and start from scratch to see if I can get better information. I'll also try and do thread dumps when I see errors.

--Tim



--
Kirill Maximov
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

Will do when upgrading to 1631.

--Tim

0
Comment actions Permalink

we are seeing the same problem with build 1654.

I have tried adding -Xmx256m settings in various places including:
- Ant build config "JVM command line params" (I can see they are used based on buildAgent/logs/teamcity.log output)
- setting JAVA_OPTS in bin/runAll.sh
- setting LAUNCHER_OPTS in buildAgent/bin/agent.sh

we are using jdk 1.5.0_08, and the build works fine with no jvm settings when running outside TeamCity.

Any help would be appreciated.
thanks
tyson

0
Comment actions Permalink

Tyson,

I'm am not sure if this is still an issue for you but I was also having memory issues today with build 1654 after I added a "tar" task to one of my build.xml targets. The solution was to add the following to the top of bin/catalina.sh:

JAVA_OPTS="$ -Xmx512m"

-Dave

0

Please sign in to leave a comment.