Collecting changes in VCS Root taking forever

Answered

We are having a problem where the "Collecting changes in VCS Root" is taking an extended length of time.  Often 4-13 minutes.  We are using github as our VCS root and have a number of roots configured.

 

Of note, we have just upgraded to 9.1.7.  I'm going to attempt to downgrade and see whether or not the problem goes away.

A few other things that I've noticed:

1. the non-teamcity server cpu usage seems to spike

2. looking at top, this seems to be java jobs that are performing git operations

3. we have mysql on the same box - but it's CPU never seems to spike

4. we have 2048MB allocated to TC Server - and sometimes it crashes with OOM exceptions

 

Here's an example of the collecting VCS changes output:

[17:25:14]Collecting changes in 1 VCS root (8m:02s)
[17:25:14][Collecting changes in 1 VCS root] VCS Root details
3 comments

I have 8 threads that are collecting VCS changes

Most are in the following stack trace:

at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:582)
at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.tryLockChangesCollecting(VcsRootInstanceImpl.java:57)
at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.tryLockChangesCollecting(VcsRootInstanceImpl.java:102)
at jetbrains.buildServer.vcs.impl.VcsChangesSyncFetcher.submit(VcsChangesSyncFetcher.java:34)
at jetbrains.buildServer.vcs.impl.VcsChangesSyncFetcher.loadChangesNoLocking(VcsChangesSyncFetcher.java:55)
at jetbrains.buildServer.vcs.impl.VcsChangesLoaderImpl.loadChangesBetweenRoots(VcsChangesLoaderImpl.java:0)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java:5)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$1.call(VcsChangesCollectorImpl.java)
at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:59)
at jetbrains.buildServer.serverSide.impl.CancelableTaskHolder.waitForTaskToComplete(CancelableTaskHolder.java:17)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.doCollectChanges(TopBuildDependencyGraphImpl.java:108)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.access$000(TopBuildDependencyGraphImpl.java:138)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:2)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$4.run(TopBuildDependencyGraphImpl.java:1)
at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:17)
at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:12)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.runAsSystem(TopBuildDependencyGraphImpl.java:131)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:9)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:65)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChangesForGraph(TopBuildDependencyGraphImpl.java:169)
at jetbrains.buildServer.serverSide.impl.BuildChangesCollector.collectChangesForBuild(BuildChangesCollector.java:15)
at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:0)
at jetbrains.buildServer.serverSide.impl.BuildStarter$2$1.run(BuildStarter.java)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:89)
at jetbrains.buildServer.serverSide.impl.BuildStarter$2.run(BuildStarter.java)

0

Here is the memory allocation & processor usage

 

0

If someone faced the same issue, please see the related issue: https://youtrack.jetbrains.com/issue/TW-45326.

0

Please sign in to leave a comment.