java.lang.OutOfMemoryError: PermGen space since upgrading to TeamCity 6.5

Since upgrading from TeamCity 6.0.3 to 6.5 we are getting consistent 'java.lang.OutOfMemoryError: PermGen' space errors during compilation while running 'mvn clean install'. I have tried changing both the server and all of our 6 agent's MaxPermSize. But it has not helped. This problem only happens when running maven commands. I noticed that the maven libraries bundled with the version of maven included with TeamCity have changed. So, I changed our build configs to use a local instance of maven. This has not helped. I am getting to the point of rolling back to 6.0.3 (if possible).

Env:
Server:
64-bit CentOS
dual-core
6GB memory

JAVA_OPTS='-Xms256m -Xmx1024m -XX:MaxPermSize=512m'
MAVEN_OPTS='-Xms256m -Xmx1024m -XX:MaxPermSize=512m'



Agents:

64-bit CentOS
dual-core
4GB memory

JAVA_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m'
MAVEN_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m'



Current settings:
Agents:
agent.sh:
TEAMCITY_AGENT_MEM_OPTS="-Xms256m -Xmx1024m -XX:MaxPermSize=512m"

Server:
teamcity-server.sh:
TEAMCITY_SERVER_MEM_OPTS="-Xms512m -Xmx2048m -XX:MaxPermSize=1024m"

These errors never occured before the upgrade. Any suggestions?

Attached is a recent thread dump.

Thanks,
-Dave

Attachment(s):
threadDump.txt.zip
8 comments
Comment actions Permalink

I tried (once again) to change settings to get past this problem, this time:

Env:
Server:
JAVA_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'
MAVEN_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'


Agents:

JAVA_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'
MAVEN_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'



Current settings:
Agents:
agent.sh:
TEAMCITY_AGENT_MEM_OPTS="-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m"

Server:
teamcity-server.sh:
TEAMCITY_SERVER_MEM_OPTS="-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m"

This helped reduce the number of PermGen failures which were happening about 50% of the time. But it still happens.
Does anyone have any suggestions?

Thanks,
-Dave

0
Comment actions Permalink

Could you please attach your build log here, or send it to teamcity-feedback[at]jetbrains.com?

0
Comment actions Permalink

Per your request, I emailed the build agent log with PermGen errors.

-Dave

0
Comment actions Permalink
Dave,
Don't use MAVEN_OPTS. Use "JVM command line parameters" in the runner page instead.
0
Comment actions Permalink

Sergey,

Just so I understand what you are suggesting, is this what you propose?:

  • In each TeamCity agent.sh file, revert the TEAMCITY_AGENT_MEM_OPTS value

        from this:
        TEAMCITY_AGENT_MEM_OPTS="-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m"
        to this:
        TEAMCITY_AGENT_MEM_OPTS="-Xmx384m"

  • In each problematic build, add this to the build runner's 'JVM command line parameters':

        -Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m

Is this correct?

BTW, yesterday I upgraded to TeamCity 6.5.1.

Thanks,
-Dave

0
Comment actions Permalink

Hi Dave,

As far as I understand the OOM is happpening inside your maven builds, not agents.

TEAMCITY_AGENT_MEM_OPTS is (and always has been) effective only for an agent's JVM. These settings aren't propogated to build process JVM like Maven, Ant and others. So altering them doesn't take any effect on builds.

In TeamCity versions prior to 6.5 Maven was executed by invoking the 'mvn' shell script. The side effect of this was that you were able specify Maven's JVM by setting arguments into MAVEN_OPTS variable.
Since version 6.5 Maven is executed by direct java command. You can see it the command line shown in the top of a build log.
This means that MAVEN_OPTS isn't effective anymore.

You should specify individual memory settings into  'JVM command line parameters' for each build that requires it.

If your agents work well with "-Xmx384m" then it would be better to set TEAMCITY_AGENT_MEM_OPTS to this value.

and yes in each problematic build you should specify enough heap/perm/permgen size.

0
Comment actions Permalink

Sergey,

Thank you for the detailed explanation. Now I understand the problem and solution.

I modified the  'JVM command line parameters' for each problematic build yesterday and the problems seem to have gone away. I will reduce the agent memory settings back to the default as it seems the increased values are unnecessary.

One comment: This has been a problem that I struggled with for weeks and it would have been very helpful if this information had been in the 'What's New in TeamCity 6.5' section of the documentation. It's an easy configuration change to make if I had known to make it. Had I known this change before upgrading to v6.5 I would have avoided adding more memory to our 6 build agents, changing the server/agents build configurations multiple times and incurring the wrath of developers who had their remote runs fail about 50% of the time.

-Dave

0
Comment actions Permalink

You're absolutely right, Dave. This is my fault I didn't foresee such problems before the release.

We're working on the corresponding section in the upgrade notes.

Thank you and sorry for inconvenience.

0

Please sign in to leave a comment.