java.lang.OutOfMemoryError: PermGen space since upgrading to TeamCity 6.5

I tried (once again) to change settings to get past this problem, this time:

Env:
Server:
JAVA_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'
MAVEN_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'


Agents:

JAVA_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'
MAVEN_OPTS='-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m'



Current settings:
Agents:
agent.sh:
TEAMCITY_AGENT_MEM_OPTS="-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m"

Server:
teamcity-server.sh:
TEAMCITY_SERVER_MEM_OPTS="-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m"

This helped reduce the number of PermGen failures which were happening about 50% of the time. But it still happens.
Does anyone have any suggestions?

Thanks,
-Dave

Could you please attach your build log here, or send it to teamcity-feedback[at]jetbrains.com?

Per your request, I emailed the build agent log with PermGen errors.

-Dave

Sergey,

Just so I understand what you are suggesting, is this what you propose?:

• In each TeamCity agent.sh file, revert the TEAMCITY_AGENT_MEM_OPTS value
  

        from this:
        TEAMCITY_AGENT_MEM_OPTS="-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m"
        to this:
        TEAMCITY_AGENT_MEM_OPTS="-Xmx384m"

• In each problematic build, add this to the build runner's 'JVM command line parameters':

        -Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=1024m

Is this correct?

BTW, yesterday I upgraded to TeamCity 6.5.1.

Thanks,
-Dave

Hi Dave,

As far as I understand the OOM is happpening inside your maven builds, not agents.

TEAMCITY_AGENT_MEM_OPTS is (and always has been) effective only for an agent's JVM. These settings aren't propogated to build process JVM like Maven, Ant and others. So altering them doesn't take any effect on builds.

In TeamCity versions prior to 6.5 Maven was executed by invoking the 'mvn' shell script. The side effect of this was that you were able specify Maven's JVM by setting arguments into MAVEN_OPTS variable.
Since version 6.5 Maven is executed by direct java command. You can see it the command line shown in the top of a build log.
This means that MAVEN_OPTS isn't effective anymore.

You should specify individual memory settings into  'JVM command line parameters' for each build that requires it.

If your agents work well with "-Xmx384m" then it would be better to set TEAMCITY_AGENT_MEM_OPTS to this value.

and yes in each problematic build you should specify enough heap/perm/permgen size.

Sergey,

Thank you for the detailed explanation. Now I understand the problem and solution.

I modified the  'JVM command line parameters' for each problematic build yesterday and the problems seem to have gone away. I will reduce the agent memory settings back to the default as it seems the increased values are unnecessary.

One comment: This has been a problem that I struggled with for weeks and it would have been very helpful if this information had been in the 'What's New in TeamCity 6.5' section of the documentation. It's an easy configuration change to make if I had known to make it. Had I known this change before upgrading to v6.5 I would have avoided adding more memory to our 6 build agents, changing the server/agents build configurations multiple times and incurring the wrath of developers who had their remote runs fail about 50% of the time.

-Dave

You're absolutely right, Dave. This is my fault I didn't foresee such problems before the release.

We're working on the corresponding section in the upgrade notes.

Thank you and sorry for inconvenience.