TeamCity V6 - OutofMemoryError on 'checking changes'

Hi Adam,
how much memory did you give to fetch process and how big is your repository?  
If you repository is quite big you may clone it using a native git and set up VCS Root to use it. In this case Teamcity will do just small fetches which should not cause OutOfMemoryErrors.

Where would I clone this to?  Cloning it to the agent work directory and naming that directory causes the directroy to be nuked.

Hi Raymond,

this error occurs only on the server, so if you get this error message you could clone your repository somewhere on the server and set a VCS root option "Clone repository to:" to the dir with the clone.
But I would first try to increase amount of memory for the fetch process using teamcity.git.fetch.process.max.memory internal option, please see http://confluence.jetbrains.net/display/TCD65/Git+%28JetBrains%29#Git%28JetBrains%29-InternalProperties for details.

I have already bumped up the settings to:

teamcity.git.fetch.process.max.memory=7168M
teamcity.git.fetch.separate.process=true

In the TeamCity/conf/internal.properties file with no luck.  This is where I would set these, correct?

Yes, this is the right place.
Is your repository so big? If so - try clone it manually and point to it in the settings of the VCS root.

The repo is around 300MB with about a billion commits over the last number of years.  Working on the alternate solution and will let you know how it goes.

Thanks!

[11:27:59]:
java.util.concurrent.ExecutionException:  jetbrains.buildServer.vcs.VcsException: Problem collecting changes for  'M2 Core :: ray_testing' : 'git fetch' command failed. stderr: java.lang.OutOfMemoryError: Java heap space      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:163)      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:118)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:561)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:544)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:507)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:462)      at org.eclipse.jgit.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:178)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:410)      at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:649)      at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:280)      at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:225)      at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:214)      at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:149)      at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:111)      at org.eclipse.jgit.transport.Transport.fetch(Transport.java:903)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.fetch(Fetcher.java:81)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.main(Fetcher.java:47) java.util.concurrent.ExecutionException:  jetbrains.buildServer.vcs.VcsException: Problem collecting changes for  'M2 Core :: ray_testing' : 'git fetch' command failed. stderr: java.lang.OutOfMemoryError: Java heap space      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:163)      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:118)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:561)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:544)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:507)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:462)      at org.eclipse.jgit.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:178)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:410)      at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:649)      at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:280)      at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:225)      at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:214)      at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:149)      at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:111)      at org.eclipse.jgit.transport.Transport.fetch(Transport.java:903)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.fetch(Fetcher.java:81)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.main(Fetcher.java:47)      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)      at java.util.concurrent.FutureTask.get(FutureTask.java:83)      at jetbrains.buildServer.serverSide.impl.CancelableTaskHolder.waitForTaskToComplete(CancelableTaskHolder.java:1)      at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.doCollectChanges(TopBuildDependencyGraphImpl.java:9)      at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.access$000(TopBuildDependencyGraphImpl.java:29)      at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:1)      at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java)      at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$4.run(TopBuildDependencyGraphImpl.java:1)      at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:21)      at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:1)      at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.runAsSystem(TopBuildDependencyGraphImpl.java:66)      at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:82)      at  jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:27)      at jetbrains.buildServer.serverSide.impl.BuildChangesCollector.collectChangesForBuild(BuildChangesCollector.java:21)      at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:18)      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)      at java.util.concurrent.FutureTask.run(FutureTask.java:138)      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)      at java.lang.Thread.run(Thread.java:619) Caused by: jetbrains.buildServer.vcs.VcsException: Problem collecting  changes for 'M2 Core :: ray_testing' : 'git fetch' command failed. stderr: java.lang.OutOfMemoryError: Java heap space      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:163)      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:118)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:561)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:544)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:507)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:462)      at org.eclipse.jgit.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:178)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:410)      at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:649)      at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:280)      at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:225)      at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:214)      at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:149)      at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:111)      at org.eclipse.jgit.transport.Transport.fetch(Transport.java:903)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.fetch(Fetcher.java:81)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.main(Fetcher.java:47)      at  jetbrains.buildServer.vcs.impl.BuildChangesLoaderContext$BuildTypeCollectorResult.reThrowProblemException(BuildChangesLoaderContext.java:40)      at  jetbrains.buildServer.vcs.impl.BuildChangesLoaderContext$BuildTypeCollectorResult.access$1100(BuildChangesLoaderContext.java:35)      at jetbrains.buildServer.vcs.impl.BuildChangesLoaderContext.checkNoProblemsInMyBuildTypes(BuildChangesLoaderContext.java:43)      at jetbrains.buildServer.vcs.impl.BuildChangesLoaderContext.checkingForChangesFinished(BuildChangesLoaderContext.java:60)      at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:4)      at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java)      ... 5 more Caused by: jetbrains.buildServer.vcs.VcsException: 'git fetch' command failed. stderr: java.lang.OutOfMemoryError: Java heap space      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:163)      at org.eclipse.jgit.storage.pack.BinaryDelta.apply(BinaryDelta.java:118)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:561)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:544)      at org.eclipse.jgit.transport.PackParser.resolveDeltas(PackParser.java:507)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:462)      at org.eclipse.jgit.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:178)      at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:410)      at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:649)      at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:280)      at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:225)      at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:214)      at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:149)      at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:111)      at org.eclipse.jgit.transport.Transport.fetch(Transport.java:903)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.fetch(Fetcher.java:81)      at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.main(Fetcher.java:47)      at jetbrains.buildServer.buildTriggers.vcs.git.CommandLineUtil.getCommandLineError(CommandLineUtil.java:41)      at jetbrains.buildServer.buildTriggers.vcs.git.FetchCommandImpl.fetchInSeparateProcess(FetchCommandImpl.java:129)      at jetbrains.buildServer.buildTriggers.vcs.git.FetchCommandImpl.fetch(FetchCommandImpl.java:68)      at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.fetch(GitVcsSupport.java:1078)      at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.fetchBranchData(GitVcsSupport.java:1060)      at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentVersion(GitVcsSupport.java:991)      at  jetbrains.buildServer.vcs.impl.VcsRootInstancesManagerImpl$SVcsRootInstance.getCurrentRevision(VcsRootInstancesManagerImpl.java:25)      at jetbrains.buildServer.vcs.impl.VcsManagerImpl.getRevisionsForAllRoots(VcsManagerImpl.java:181)      at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:709)      at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:64)      at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$2.call(VcsChangesCollectorImpl.java)      at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$2.call(VcsChangesCollectorImpl.java:1)

Hi Raymond,

it is really strange that you have OutOfMemoryError with your settings. Stacktrace shows that error occurs during parsing pack file, received from remote repository. Since you cloned repository manually, this pack file should be small and should not cause any problems. Did make a bare clone of your repository?

I had not -- I made a regular clone.  Trying it bare now.

It's been 'checking changes' now for nearly an hour.  I'll let it run a while and see what it does.  With this repo it takes a long time even with git at the CLI -- though not nearly this long, 5 minutes or so -- so we'll see how it goes.  At least the memory issue seems to be resolved.  I'll let you know if there are more issues, but in the mean time, thank you for your help!

Hi Raymond,

did 'checking for changes' process finished? I can think of one reason of why it is so slow: submodules. Do you use them?

No submodules.  The bare clone did the trick, altough it ran about 4 hours before I killed it, started it again and it came through finally about a half hour later.

I then ran into more memory issues running another project that depended on the first (Both Rails 3 on jRuby).  I was getting a GC Memory error when running rake:integration tests, using Selenium and a headless XTerm/Firefox setup.  When I finally gave up the rake step and instead used a CLI step, calling 'ruby -XmxXXX -S rake', bumping up the available memory to the Rails app I got it to run finally.  jRuby 1.6.2/RVM setup...

Thank you for checking back.  This was probably the toughest install of anything that I've done in 20 years!  Next:  MSDev on a Blackberry =)  I am looking forward to using TC -- it seems well thought out.

Raymond,
I think checking for changes shouldn't be so long. If you will have this problem once again, please take few thread dumps (you can do it on the Administration > Server Configuration > Diagnostics) and provide a teamcity-vcs.log.
Also I fixed a few more bugs recently, you can take the last build of git-plugin with fixes from http://teamcity.jetbrains.com/viewType.html?buildTypeId=bt245&tab=buildTypeStatusDiv.