java.lang.OutOfMemoryError: Java heap space Follow
I got this error before, and fixed it by increasing the memory pool size from 512mb to 1024mb in the Tomcat6w Teamcity Web Server Properties under the Java tab,
Our server has 32GB of Ram, and we are using Perforce as our VCS, and Teamcity version 4.5.2 with 10 agents. I tried to increase it to 2Gb, but the server won't start instead I increased it to 1535mb. Is there something else that I should set?
java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205) at java.util.concurrent.FutureTask.get(FutureTask.java:80) at jetbrains.buildServer.serverSide.impl.CancelableTaskHolder.waitForTaskToComplete(CancelableTaskHolder.java:14) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.doCollectChanges(TopBuildDependencyGraphImpl.java:41) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.access$000(TopBuildDependencyGraphImpl.java:36) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:1) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$4.run(TopBuildDependencyGraphImpl.java:1) at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:7) at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:29) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.runAsSystem(TopBuildDependencyGraphImpl.java:85) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:37) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:32) at jetbrains.buildServer.serverSide.impl.BuildChangesCollector.collectChangesForBuild(BuildChangesCollector.java:12) at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:7) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'AotD Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.rethrowIfOurBuildType(VcsChangesCollectorImpl.java:77) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.checkNoProblemsInMyBuildTypes(VcsChangesCollectorImpl.java:72) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.checkingForChangesFinished(VcsChangesCollectorImpl.java:106) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:0) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:3) ... 5 more Caused by: jetbrains.buildServer.vcs.VcsException: Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:175) at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:10) at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:51) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1) ... 5 more Caused by: java.lang.OutOfMemoryError: Java heap space at java.lang.String.<init>(String.java:208) at java.lang.StringBuilder.toString(StringBuilder.java:431) at com.intellij.openapi.util.text.StringUtil.convertLineSeparators(StringUtil.java:239) at com.intellij.openapi.util.text.StringUtil.convertLineSeparators(StringUtil.java:210) at jetbrains.buildServer.StreamGobbler.getReadString(StreamGobbler.java:84) at jetbrains.buildServer.ExecResult.getStdout(ExecResult.java:70) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:187) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:209) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.describeChangeList(PerforceConnection.java:83) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.loadChanges(PerforceSupport.java:273) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.getChanges(PerforceSupport.java:255) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.collectChanges(PerforceSupport.java:213) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.collectBuildChanges(PerforceSupport.java:177) at jetbrains.buildServer.vcs.VcsSupport.collectChanges(VcsSupport.java:50) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges(VcsChangesLoader.java:134) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:10) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChangesForRule(VcsChangesLoader.java:163) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges4CommonCheckoutRule(VcsChangesLoader.java:31) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:99) ... 9 more
Our server has 32GB of Ram, and we are using Perforce as our VCS, and Teamcity version 4.5.2 with 10 agents. I tried to increase it to 2Gb, but the server won't start instead I increased it to 1535mb. Is there something else that I should set?
java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205) at java.util.concurrent.FutureTask.get(FutureTask.java:80) at jetbrains.buildServer.serverSide.impl.CancelableTaskHolder.waitForTaskToComplete(CancelableTaskHolder.java:14) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.doCollectChanges(TopBuildDependencyGraphImpl.java:41) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.access$000(TopBuildDependencyGraphImpl.java:36) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:1) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$4.run(TopBuildDependencyGraphImpl.java:1) at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:7) at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:29) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.runAsSystem(TopBuildDependencyGraphImpl.java:85) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:37) at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:32) at jetbrains.buildServer.serverSide.impl.BuildChangesCollector.collectChangesForBuild(BuildChangesCollector.java:12) at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:7) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'AotD Nightly and Deployment :: Nightly Clean Win32' : Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.rethrowIfOurBuildType(VcsChangesCollectorImpl.java:77) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.checkNoProblemsInMyBuildTypes(VcsChangesCollectorImpl.java:72) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$ChangesCollectorResultImpl.checkingForChangesFinished(VcsChangesCollectorImpl.java:106) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:0) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:3) ... 5 more Caused by: jetbrains.buildServer.vcs.VcsException: Error collecting changes for VCS root 'perforce: perforce1-van:1666' #31 java.lang.OutOfMemoryError: Java heap space at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:175) at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:10) at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:51) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java) at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1) ... 5 more Caused by: java.lang.OutOfMemoryError: Java heap space at java.lang.String.<init>(String.java:208) at java.lang.StringBuilder.toString(StringBuilder.java:431) at com.intellij.openapi.util.text.StringUtil.convertLineSeparators(StringUtil.java:239) at com.intellij.openapi.util.text.StringUtil.convertLineSeparators(StringUtil.java:210) at jetbrains.buildServer.StreamGobbler.getReadString(StreamGobbler.java:84) at jetbrains.buildServer.ExecResult.getStdout(ExecResult.java:70) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:187) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.runCommand(PerforceConnection.java:209) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceConnection.describeChangeList(PerforceConnection.java:83) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.loadChanges(PerforceSupport.java:273) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.getChanges(PerforceSupport.java:255) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.collectChanges(PerforceSupport.java:213) at jetbrains.buildServer.buildTriggers.vcs.perforce.PerforceSupport.collectBuildChanges(PerforceSupport.java:177) at jetbrains.buildServer.vcs.VcsSupport.collectChanges(VcsSupport.java:50) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges(VcsChangesLoader.java:134) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:10) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChangesForRule(VcsChangesLoader.java:163) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectChanges4CommonCheckoutRule(VcsChangesLoader.java:31) at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChangesForRoot(VcsChangesLoader.java:99) ... 9 more
Please sign in to leave a comment.
I also noticed in the server, whenever that happens there's an instance of perforce in Tomcat6.exe that seems to be frozen, and when I kill that process of p4 it starts to work again... It's been occuring for about at least 4 or more times a day. I'm not sure what's wrong with it.
Hello Lawrence,
Could you please take a look what is the command line for this p4 process? On Windows, you can use process explorer utility to view this:
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
Kind regards,
KIR
Yes, that's the program I was using to find out the there was instances of P4 that was hung.
p4 -u builduser -p perforce1-van:1666 describe 373306
sometimes it shows the password as well...
Hi,
How many files are there in the change 373306? How many output would produce this command if to redirect its output to a file?
Regards,
KIR
Around 80 files, and I haven't tried outputing this command into file, but I beleive it would probably produce one. I just ran the command and piped it to a txt file, the file size went to 135.142 Mb with 3389537 lines.
The file actually went to check the file size again it was actuallt 186 Mb, I realize it's doing a diff with some file that aren't needed to so one of our team member is changing them to binary so that p4 doesn't diff them. However, I was wondering if there was a way to shorten describe so that is doesn't diff the files, like running p4 with the -s parameter, which does exactly that. Is there a way to configure TeamCity to use this –s option and thus reduce the size of the p4 describe output?
Hello Lawrence,
I believe, your observation is right to the point. The code was written quite some time ago, and it looks it really should have contained -s switch.
I also think that this may be the reason for the bug TW-8162.
I've created a patch for the problem and attached it to this message. The patch should be added to the directory
TeamCity/webapps/ROOT/WEB-INF/plugins/perforce/server/ (it replaces some classes from perforce.jar located in this directory).
Could you please try this patch and let me know if it works for you.
Thanks!
KIR
Attachment(s):
a_patch.jar
Thanks Kirill, I added the patch to the specified directory and restarted the Teamcity server service. I'll let you know how it goes later today. I have a good feeling it'll fix it .
I keep fingers crossed
So far aftering running it for about 4 hour, no issues.
I'll let it run over the weekend and see.
No Issues over the weekend , seems to be fixed after the patch. Thank you. Cheers.
Thanks Lawrence, the patch is integrated to the next 4.5.x update.
Regards,
KIR