java.lang.OutOfMemoryError: Java heap space

I got this error before, and fixed it by increasing the memory pool size from 512mb to 1024mb in the Tomcat6w Teamcity Web Server Properties under the Java tab,

Our server has 32GB of Ram, and we are using Perforce as our VCS, and Teamcity version 4.5.2 with 10 agents. I tried to increase it to 2Gb, but the server won't start instead I increased it to 1535mb.  Is there something else that I should set?

java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205) at java.util.concurrent.FutureTask.get(FutureTask.java:80) at jetbrains.buildServer.serverSide.impl.CancelableTaskHolder.waitForTaskToComplete(CancelableTaskHolder.java:14) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.doCollectChanges(TopBuildDependencyGraphImpl.java:41) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.access$000(TopBuildDependencyGraphImpl.java:36) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:1) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$4.run(TopBuildDependencyGraphImpl.java:1) at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:7) at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:29) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.runAsSystem(TopBuildDependencyGraphImpl.java:85) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:37) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:32) at jetbrains.buildServer.serverSide.impl.BuildChangesCollector.collectChangesForBuild(BuildChangesCollector.java:12) at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:7) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'AotD Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.rethrowIfOurBuildType(VcsChangesCollectorImpl.java:77) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.checkNoProblemsInMyBuildTypes(VcsChangesCollectorImpl.java:72) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.checkingForChangesFinished(VcsChangesCollectorImpl.java:106) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:0) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:3) ... 5 more Caused by: jetbrains.buildServer.vcs.VcsException: Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:175) at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:10) at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:51) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1) ... 5 more Caused by: java.lang.OutOfMemoryError: Java heap space at java.lang.String.<init>(String.java:208) at java.lang.StringBuilder.toString(StringBuilder.java:431) at com.intellij.openapi.util.text.StringUtil.convertLineSeparators(StringUtil.java:239) at com.intellij.openapi.util.text.StringUtil.convertLineSeparators(StringUtil.java:210) at jetbrains.buildServer.StreamGobbler.getReadString(StreamGobbler.java:84) at jetbrains.buildServer.ExecResult.getStdout(ExecResult.java:70) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:187) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:209) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.describeChangeList(PerforceConnection.java:83) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.loadChanges(PerforceSupport.java:273) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.getChanges(PerforceSupport.java:255) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.collectChanges(PerforceSupport.java:213) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.collectBuildChanges(PerforceSupport.java:177) at jetbrains.buildServer.vcs.VcsSupport.collectChanges(VcsSupport.java:50) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges(VcsChangesLoader.java:134) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:10) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChangesForRule(VcsChangesLoader.java:163) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges4CommonCheckoutRule(VcsChangesLoader.java:31) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:99) ... 9 more



12 comments
Comment actions Permalink

I also noticed in the server, whenever that happens there's an instance of perforce in Tomcat6.exe that seems to be frozen, and when I kill that process of p4 it starts to work again...  It's been occuring for about at least 4 or more times a day.  I'm not sure what's wrong with it.

0
Comment actions Permalink

Hello Lawrence,

  Could you please take a look what is the command line for this p4 process? On Windows, you can use process explorer utility to view this:
  http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

  Kind regards,
  KIR

0
Comment actions Permalink

Yes, that's the program I was using to find out the there was instances of P4 that was hung. 

p4 -u builduser -p perforce1-van:1666 describe 373306
sometimes it shows the password as well...

0
Comment actions Permalink

Hi,

  How many files are there in the change 373306? How many output would produce this command if to redirect its output to a file?

  Regards,
  KIR

0
Comment actions Permalink

Around 80 files, and I haven't tried outputing this command into file, but I beleive it would probably produce one.  I just ran the command and piped it to a txt file, the file size went to 135.142 Mb with 3389537 lines.

0
Comment actions Permalink

The file actually went to check the file size again it was actuallt 186 Mb, I realize it's doing a diff with some file that aren't needed to so one of our team member is changing them to binary so that p4 doesn't diff them.  However, I was wondering if there was a way to shorten describe so that is doesn't diff the files, like running p4 with the -s parameter, which does exactly that.  Is there a way to configure TeamCity to use this –s option and thus reduce the size of the p4 describe output?

0
Comment actions Permalink

Hello Lawrence,

   I believe, your observation is right to the point. The code was written quite some time ago, and it looks it really should have contained -s switch.
   I also think that this may be the reason for the bug TW-8162.

   I've created a patch for the problem and attached it to this message. The patch should be added to the directory
   TeamCity/webapps/ROOT/WEB-INF/plugins/perforce/server/ (it replaces some classes from perforce.jar located in this directory).

   Could you please try this patch and let me know if it works for you.

   Thanks!
   KIR



Attachment(s):
a_patch.jar
0
Comment actions Permalink

Thanks Kirill, I added the patch to the specified directory and restarted the Teamcity server service.  I'll let you know how it goes later today.  I have a good feeling it'll fix it .

0
Comment actions Permalink

I keep fingers crossed

0
Comment actions Permalink

So far aftering running it for about 4 hour, no issues.

I'll let it run over the weekend and see.

0
Comment actions Permalink

No Issues over the weekend , seems to be fixed after the patch.  Thank you. Cheers.

0
Comment actions Permalink

Thanks Lawrence, the patch is integrated to the next 4.5.x update.

Regards,
KIR

0

Please sign in to leave a comment.