TeamCity creates a large number of threads

Avatar
Blazej Filipiak
Created

Hi,

For some reason TC constantly creates new threads and is not able to kill the previous ones. This results in reaching the thread limit (around ~4900) after that, main functionalities of TC are blocked.

 

Thread number from diagnostics:

 

Problem notifucations:

 

 

VCS problem: abt_pako_web

Edit this VCS root...
Failed for the root '"abt_pako_web" {instance id=18, parent internal id=9, parent id=AmbFwSw_AbtPako_AbtPakoWeb, description: "repos:abt_pako_web.git#refs/heads/devel"}: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

 

jetbrains.buildServer.vcs.VcsRootVcsException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.createRootException(CurrentStateCache.java:73)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentState(CurrentStateCache.java:23)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentState(VcsChangesStatesCollector.java:73)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$null$1(VcsChangesStatesCollector.java:22)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$getCollectStatesTasks$2(VcsChangesStatesCollector.java:75)
	at jetbrains.buildServer.vcs.impl.ImmediateFutureExecService$2.call(ImmediateFutureExecService.java:3)
	at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:63)
	at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:73)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.collectStatesForAllRoots(VcsChangesStatesCollector.java:27)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentStateSnapshot(VcsChangesStatesCollector.java:57)
	at jetbrains.buildServer.vcs.impl.VcsChangesFetcher.getStateAndCreateChangesCollectingTasks(VcsChangesFetcher.java:60)
	at jetbrains.buildServer.vcs.impl.VcsChangesLoader.doLoadChanges(VcsChangesLoader.java:139)
	at jetbrains.buildServer.vcs.impl.VcsChangesLoader.tryLoadChanges(VcsChangesLoader.java:48)
	at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction$1.run(VcsModificationChecker.java:5)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
	at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction.run(VcsModificationChecker.java:4)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: jetbrains.buildServer.vcs.VcsException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	... 21 more
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:798)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1343)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at org.eclipse.jgit.util.io.IsolatedOutputStream.execute(IsolatedOutputStream.java:168)
	at org.eclipse.jgit.util.io.IsolatedOutputStream.flush(IsolatedOutputStream.java:82)
	at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:143)
	at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
	at org.eclipse.jgit.transport.JschSession$JschProcess.closeOutputStream(JschSession.java:157)
	at org.eclipse.jgit.transport.JschSession$JschProcess.destroy(JschSession.java:213)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.close(TransportGitSsh.java:289)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:269)
	at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:144)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:188)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:176)
	at jetbrains.buildServer.buildTriggers.vcs.git.Retry.retry(Retry.java:58)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.lambda$getRemoteRefsJGit$5(GitRepoOperationsImpl.java:176)
	at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:30)
	at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:19)
	at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkCall(IOGuardInitializer.java:10)
	at jetbrains.buildServer.serverSide.IOGuard.allowNetworkCall(IOGuard.java:82)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.getRemoteRefsJGit(GitRepoOperationsImpl.java:174)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.lambda$lsRemoteCommand$2(GitRepoOperationsImpl.java:154)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.doGetRemoteRefs(GitVcsSupport.java:545)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.doGetRemoteRefs(GitVcsSupport.java:529)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getRemoteRefs(GitVcsSupport.java:520)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.lambda$getCurrentState$0(GitVcsSupport.java:183)
	at jetbrains.buildServer.buildTriggers.vcs.git.RepositoryManagerImpl.runWithDisabledRemove(RepositoryManagerImpl.java:256)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:179)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:174)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy.getCurrentState(GitCollectChangesPolicy.java:172)
	at jetbrains.vcs.api.services.impl.RepositoryStateServiceProvider$1.getCurrentState(RepositoryStateServiceProvider.java:5)
	at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.lambda$getCurrentState$2(VcsRootInstanceImpl.java:169)
	at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:16)
	at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:12)
	at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkAndCommandLine(IOGuardInitializer.java:6)
	at jetbrains.buildServer.serverSide.IOGuard.allowNetworkAndCommandLine(IOGuard.java:117)
	at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.getCurrentState(VcsRootInstanceImpl.java:27)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:4)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:2)
	at jetbrains.buildServer.vcs.VcsOperationProgressProviderImpl.runWithProgress(VcsOperationProgressProviderImpl.java:29)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$1.doGetCurrentState(VcsChangesStatesCollector.java:2)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentStateFuture(CurrentStateCache.java:22)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentState(CurrentStateCache.java:85)
	... 19 more
jetbrains.buildServer.vcs.VcsException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.createRootException(CurrentStateCache.java:73)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentState(CurrentStateCache.java:23)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentState(VcsChangesStatesCollector.java:73)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$null$1(VcsChangesStatesCollector.java:22)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$getCollectStatesTasks$2(VcsChangesStatesCollector.java:75)
	at jetbrains.buildServer.vcs.impl.ImmediateFutureExecService$2.call(ImmediateFutureExecService.java:3)
	at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:63)
	at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:73)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.collectStatesForAllRoots(VcsChangesStatesCollector.java:27)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentStateSnapshot(VcsChangesStatesCollector.java:57)
	at jetbrains.buildServer.vcs.impl.VcsChangesFetcher.getStateAndCreateChangesCollectingTasks(VcsChangesFetcher.java:60)
	at jetbrains.buildServer.vcs.impl.VcsChangesLoader.doLoadChanges(VcsChangesLoader.java:139)
	at jetbrains.buildServer.vcs.impl.VcsChangesLoader.tryLoadChanges(VcsChangesLoader.java:48)
	at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction$1.run(VcsModificationChecker.java:5)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
	at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction.run(VcsModificationChecker.java:4)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:798)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1343)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at org.eclipse.jgit.util.io.IsolatedOutputStream.execute(IsolatedOutputStream.java:168)
	at org.eclipse.jgit.util.io.IsolatedOutputStream.flush(IsolatedOutputStream.java:82)
	at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:143)
	at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
	at org.eclipse.jgit.transport.JschSession$JschProcess.closeOutputStream(JschSession.java:157)
	at org.eclipse.jgit.transport.JschSession$JschProcess.destroy(JschSession.java:213)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.close(TransportGitSsh.java:289)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:269)
	at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:144)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:188)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:176)
	at jetbrains.buildServer.buildTriggers.vcs.git.Retry.retry(Retry.java:58)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.lambda$getRemoteRefsJGit$5(GitRepoOperationsImpl.java:176)
	at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:30)
	at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:19)
	at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkCall(IOGuardInitializer.java:10)
	at jetbrains.buildServer.serverSide.IOGuard.allowNetworkCall(IOGuard.java:82)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.getRemoteRefsJGit(GitRepoOperationsImpl.java:174)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.lambda$lsRemoteCommand$2(GitRepoOperationsImpl.java:154)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.doGetRemoteRefs(GitVcsSupport.java:545)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.doGetRemoteRefs(GitVcsSupport.java:529)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getRemoteRefs(GitVcsSupport.java:520)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.lambda$getCurrentState$0(GitVcsSupport.java:183)
	at jetbrains.buildServer.buildTriggers.vcs.git.RepositoryManagerImpl.runWithDisabledRemove(RepositoryManagerImpl.java:256)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:179)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:174)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy.getCurrentState(GitCollectChangesPolicy.java:172)
	at jetbrains.vcs.api.services.impl.RepositoryStateServiceProvider$1.getCurrentState(RepositoryStateServiceProvider.java:5)
	at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.lambda$getCurrentState$2(VcsRootInstanceImpl.java:169)
	at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:16)
	at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:12)
	at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkAndCommandLine(IOGuardInitializer.java:6)
	at jetbrains.buildServer.serverSide.IOGuard.allowNetworkAndCommandLine(IOGuard.java:117)
	at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.getCurrentState(VcsRootInstanceImpl.java:27)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:4)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:2)
	at jetbrains.buildServer.vcs.VcsOperationProgressProviderImpl.runWithProgress(VcsOperationProgressProviderImpl.java:29)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$1.doGetCurrentState(VcsChangesStatesCollector.java:2)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentStateFuture(CurrentStateCache.java:22)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentState(CurrentStateCache.java:85)
	... 19 more
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Thread.java:798)
	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1343)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at org.eclipse.jgit.util.io.IsolatedOutputStream.execute(IsolatedOutputStream.java:168)
	at org.eclipse.jgit.util.io.IsolatedOutputStream.flush(IsolatedOutputStream.java:82)
	at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:143)
	at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:182)
	at org.eclipse.jgit.transport.JschSession$JschProcess.closeOutputStream(JschSession.java:157)
	at org.eclipse.jgit.transport.JschSession$JschProcess.destroy(JschSession.java:213)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.close(TransportGitSsh.java:289)
	at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:269)
	at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:144)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:188)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:176)
	at jetbrains.buildServer.buildTriggers.vcs.git.Retry.retry(Retry.java:58)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.lambda$getRemoteRefsJGit$5(GitRepoOperationsImpl.java:176)
	at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:30)
	at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:19)
	at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkCall(IOGuardInitializer.java:10)
	at jetbrains.buildServer.serverSide.IOGuard.allowNetworkCall(IOGuard.java:82)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.getRemoteRefsJGit(GitRepoOperationsImpl.java:174)
	at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.lambda$lsRemoteCommand$2(GitRepoOperationsImpl.java:154)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.doGetRemoteRefs(GitVcsSupport.java:545)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.doGetRemoteRefs(GitVcsSupport.java:529)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getRemoteRefs(GitVcsSupport.java:520)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.lambda$getCurrentState$0(GitVcsSupport.java:183)
	at jetbrains.buildServer.buildTriggers.vcs.git.RepositoryManagerImpl.runWithDisabledRemove(RepositoryManagerImpl.java:256)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:179)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:174)
	at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy.getCurrentState(GitCollectChangesPolicy.java:172)
	at jetbrains.vcs.api.services.impl.RepositoryStateServiceProvider$1.getCurrentState(RepositoryStateServiceProvider.java:5)
	at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.lambda$getCurrentState$2(VcsRootInstanceImpl.java:169)
	at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:16)
	at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:12)
	at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkAndCommandLine(IOGuardInitializer.java:6)
	at jetbrains.buildServer.serverSide.IOGuard.allowNetworkAndCommandLine(IOGuard.java:117)
	at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.getCurrentState(VcsRootInstanceImpl.java:27)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:4)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:2)
	at jetbrains.buildServer.vcs.VcsOperationProgressProviderImpl.runWithProgress(VcsOperationProgressProviderImpl.java:29)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$1.doGetCurrentState(VcsChangesStatesCollector.java:2)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentStateFuture(CurrentStateCache.java:22)
	at jetbrains.buildServer.vcs.impl.CurrentStateCache.getCurrentState(CurrentStateCache.java:85)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentState(VcsChangesStatesCollector.java:73)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$null$1(VcsChangesStatesCollector.java:22)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$getCollectStatesTasks$2(VcsChangesStatesCollector.java:75)
	at jetbrains.buildServer.vcs.impl.ImmediateFutureExecService$2.call(ImmediateFutureExecService.java:3)
	at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:63)
	at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:73)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.collectStatesForAllRoots(VcsChangesStatesCollector.java:27)
	at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentStateSnapshot(VcsChangesStatesCollector.java:57)
	at jetbrains.buildServer.vcs.impl.VcsChangesFetcher.getStateAndCreateChangesCollectingTasks(VcsChangesFetcher.java:60)
	at jetbrains.buildServer.vcs.impl.VcsChangesLoader.doLoadChanges(VcsChangesLoader.java:139)
	at jetbrains.buildServer.vcs.impl.VcsChangesLoader.tryLoadChanges(VcsChangesLoader.java:48)
	at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction$1.run(VcsModificationChecker.java:5)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
	at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction.run(VcsModificationChecker.java:4)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

 

Service thread limit:

$ cat /sys/fs/cgroup/pids/system.slice/teamcity.service/pids.max
4915

 

From catalina logs:

10-Aug-2022 12:42:08.947 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [sshd-SshClient[3da9582f]-nio2-thread-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:829)
10-Aug-2022 12:42:08.956 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [sshd-SshClient[13dbb8c3]-timer-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
 java.base@11.0.13/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:829)
10-Aug-2022 12:42:08.965 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-1549] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/sun.nio.ch.EPoll.wait(Native Method)
 java.base@11.0.13/sun.nio.ch.EPollPort$EventHandlerTask.poll(EPollPort.java:200)
 java.base@11.0.13/sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:281)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:829)
10-Aug-2022 12:42:08.974 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [sshd-SshClient[13dbb8c3]-nio2-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:829)
10-Aug-2022 12:42:08.982 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [sshd-SshClient[13dbb8c3]-nio2-thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:829)
10-Aug-2022 12:42:08.991 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [sshd-SshClient[13dbb8c3]-nio2-thread-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:829)
10-Aug-2022 12:42:09.000 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [sshd-SshClient[13dbb8c3]-nio2-thread-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base@11.0.13/jdk.internal.misc.Unsafe.park(Native Method)
 java.base@11.0.13/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
 java.base@11.0.13/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
 java.base@11.0.13/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
 java.base@11.0.13/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 java.base@11.0.13/java.lang.Thread.run(Thread.java:829)

Thread dump & current logs:

Upload id: 2022_08_11_22ESgjTNEN45EWEHbxU1Ct (files: 2022-08-11_10.29.28-manual.txt, teamcity_server_logs_recent_2022-08-11_1031.zip)

 

Any ideas?

 

Votes

2

Share

5 comments
Avatar
Anthony Timson
Created

Thank God! I have had this problem for two weeks and despite searching for hours am no closer to finding anyone else with the issue until now.

Our three agent system that has worked well for many years started going wrong when we upgraded the OS (yum update), changed from Java8 to Java11 (openjdk), and updated TeamCity. Since then TeamCity has updated again (108706).

Everything works fine for a couple of hours then BANG. Nothing. Out of Memory Errors as per your post, but there is plenty of memory. What there is also are hundreds of SVN connections open - we're using SVN not GIT, we are old :P. We have one VCSROOT and one SVN server.

My buildserver is currently dead and I have 3251 threads of sshd-SshClient running.

0
Avatar
Anthony Timson
Created

You might want to try a "ps -e -T" when your system is hung, then you'll know what process is killing it.

0
Avatar
Blazej Filipiak
Created

Ok, so we are not alone that's a start ;) but how do we fix this?

$ ps -e -T | grep sshd-SshClient | wc -l
3991
0
Avatar
Anthony Timson
Created

We have paid support so I thought I would ask them – here is the response (I have attached the svn.zip):

@Ant sorry for the delay. It seems that we already have the fix for the issue. In the related issue you can find the svn.zip plugin with the fix. Could you please install the plugin to the server and check if it helps in your case?

https://app.box.com/s/j0ltlcn5h8udlaa0p097h48mt9zuryv3

1
Avatar
Blazej Filipiak
Created

Ok, so our TC is running for ~20h now and so far so good.

0

Please sign in to leave a comment.