Performance Problems TeamCity Server

Permanently deleted user
Created January 22, 2014 15:52
In the last few days we have encountered some Troubles with our TeamCity Server.
After some time our Server needs a lot processor performance and TeamCity cannot be accessed.

Some points to our setup:
- TeamCity runs on a headless ubuntu VM.
- Problem exists in Version 8.0.2 and 8.0.6
- We run Youtrack (Version 5.0.6) on the same server.
- Our Build Agent runs on a seperate windows XP VM.

with the top command we found the process that is using so much CPU
then we used ps x command to find out what commands the process is running with the folowing output, letting us suggest that something is wrong with the TeamCity-Server:

28343 ? Sl 68:57 /usr/lib/jvm/java-6-openjdk-amd64/bin/java -Djava.util.logging.config.file=/usr/share/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xmx512m -XX:MaxPermSize=270m -Dlog4j.configuration=file:/usr/share/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djava.awt.headless=true -Djava.endorsed.dirs=/usr/sha

In our teamcity-server.log we find the folowing waring:

[2014-01-22 14:58:54,110] WARN - jetbrains.buildServer.SERVER - Average TeamCity server memory usage for 'PS Old Gen' pool during the last 5 minutes is 96.5% of maximum available memory: 341.37 MB. This can cause significant server slowdown. Please review TeamCity <a href='http://www.jetbrains.net/confluence/display/TCD8/Reporting+Issues#ReportingIssues-ReportingIssues-OutOfMemoryProblems' target='_blank'>documentation</a> for possible solutions.

Stopping and starting the Server usually has no effect or we get some other strange exceptions afterwards .Usually we have to reboot our VM to get TeamCity and Youtrack back online.

Unfortunatly I couldn't find out a solution for this Problem yet. Does anyone have any suggestions why our Server dayly stops working and uses that high performance?
4 comments
Pavel Sher
Created January 22, 2014 15:58
According to error message you need to increase amount of memory available to TeamCity, i.e. Xmx parameter.
Permanently deleted user
Created January 27, 2014 07:58
Thanks for your help so far. Could you explain me how to increase the Xmx parameter?
Pavel Sher
Created January 27, 2014 09:03
You can set environment variable TEAMCITY_SERVER_MEM_OPTS to something like this:
TEAMCITY_SERVER_MEM_OPTS=-Xmx768m -XX:MaxPermSize=270m

Then restart server. If everything is done correctly, you should see changed arguments on Administration -> Diagnostics page.
Permanently deleted user
Created April 20, 2017 19:30
I am also facing same error , here is heap log
TeamCity Professional 10.0.4 (build 42538)
OS: Windows Server 2012 R2, version 6.3, x86
Java: 1.8.0_66, Java HotSpot(TM) Server VM (32 bit) (25.66-b17, mixed mode), Java(TM) SE Runtime Environment (1.8.0_66-b17), Oracle Corporation
JVM args: -Djava.util.logging.config.file=D:\TeamCity\bin\..\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xrs -Xmx512m -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j.configuration=file:../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djava.endorsed.dirs=D:\TeamCity\bin\..\endorsed -Dcatalina.base=D:\TeamCity\bin\.. -Dcatalina.home=D:\TeamCity\bin\.. -Djava.io.tmpdir=D:\TeamCity\bin\..\temp
Database type: MSSQL
Current OS user: svc_tcs

2017-04-20 15:25:37
Full thread dump Java HotSpot(TM) Server VM (25.66-b17 mixed mode):

"http-nio-95-Acceptor-0" daemon group="main" prio=5 tid=22 nid=22 runnable 
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:825)
    at java.lang.Thread.run(Thread.java:745)

"RMI RenewClean-[localhost:49266]" daemon group="system" prio=5 tid=76 nid=76 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:553)
    at java.lang.Thread.run(Thread.java:745)

"TC: 15:25:17 Processing watcher Files watcher jetbrains.buildServer.web.impl.WatchingResourceUnpackerState$1@1aab452 with state 544 files: "D:\TeamCity\webapps\ROOT\WEB-INF\plugins\.unpacked\jetbrains.git\server\JavaEWAH-0.7.9.jar" and others; 15:25:17 Task started; FilesWatcher 1" daemon group="main" prio=5 tid=46 nid=46 runnable 
   java.lang.Thread.State: RUNNABLE
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.exists(File.java:819)
    at jetbrains.buildServer.configuration.WatchedFileState.isChanged(WatchedFileState.java:53)
    at jetbrains.buildServer.configuration.FilesState.computeChanges(FilesState.java:71)
    at jetbrains.buildServer.configuration.FilesWatcher.detectChanges(FilesWatcher.java:322)
    at jetbrains.buildServer.configuration.FilesWatcher$1.run(FilesWatcher.java:284)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Build estimates updater 1" daemon group="main" prio=5 tid=161 nid=161 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.timeEstimation.CachingBuildEstimator$2.run(CachingBuildEstimator.java:15)
    at java.lang.Thread.run(Thread.java:745)

"OSProcessHandler pooled thread" daemon group="main" prio=10 tid=71 nid=71 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at com.intellij.execution.process.OSProcessHandler$ReadProcessThread.readNextByte(OSProcessHandler.java:305)
    at com.intellij.execution.process.OSProcessHandler$ReadProcessThread.run(OSProcessHandler.java:275)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Reference Handler" daemon group="system" prio=10 tid=2 nid=2 waiting 
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)

"Normal executor 4" group="main" prio=5 tid=165 nid=165 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"net.sf.ehcache.CacheManager@125b7ac" daemon group="main" prio=5 tid=48 nid=48 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

"Build Triggers Monitor 1" daemon group="main" prio=5 tid=172 nid=172 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:8)
    at java.lang.Thread.run(Thread.java:745)

"http-nio-95-exec-1" daemon group="main" prio=5 tid=24 nid=24 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"Normal executor 2" group="main" prio=5 tid=155 nid=155 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"TC: 15:23:37 Collect changes "PFI700" {instance id=20, parent internal id=13, parent id=Pfi700_Pfi700, description: "vault: http://pr1pscmvault01.nj.pfs.net/VaultService"}; 15:23:37 Loading VCS changes for "PFI700" {instance id=20, parent internal id=13, parent id=Pfi700_Pfi700, description: "vault: http://pr1pscmvault01.nj.pfs.net/VaultService"}; 15:23:37 Task started; VCS Periodical executor 1" group="main" prio=5 tid=173 nid=173 runnable 
   java.lang.Thread.State: RUNNABLE
    at system.DateTime.<init>(_VMW_Extended_DateTime.java:61422)
    at system.DateTime.__Clone__(_VMW_Extended_DateTime.java:61422)
    at system.DateTime.get_Now(_VMW_Extended_DateTime.java:113)
    at VaultLib.VaultDateTime.get_Now(Unknown Source)
    at VaultClientOperationsLib.VaultClientFile.<init>(Unknown Source)
    at VaultClientOperationsLib.VaultClientFile.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.VaultClientFolder.Clone(Unknown Source)
    at VaultClientOperationsLib.PendingChangeRepository.<init>(Unknown Source)
    at VaultClientOperationsLib.TreeCache.access$1800(Unknown Source)
    at VaultClientOperationsLib.TreeCache.GetPendingChangeRepository(Unknown Source)
    at VaultClientOperationsLib.TreeCache.get_Repository(Unknown Source)
    at VaultClientOperationsLib.ClientInstance.SetActiveRepositoryID(Unknown Source)
    at VaultClientOperationsLib.ClientInstance.SetActiveRepositoryID(Unknown Source)
    at VaultClientIntegrationLib.ServerOperations.SetRepository(ServerOperations.cs:4281)
    at VaultClientIntegrationLib.ServerOperations.SetRepository(ServerOperations.cs:4251)
    at VaultClientIntegrationLib.ServerOperations.Login(ServerOperations.cs:4181)
    at VaultClientIntegrationLib.ServerOperations.Login(ServerOperations.cs:4301)
    at jetbrains.buildServer.buildTriggers.vcs.vault.impl.VaultConnectionImpl.login(VaultConnectionImpl.java:249)
    at jetbrains.buildServer.buildTriggers.vcs.vault.impl.VaultConnectionImpl.login(VaultConnectionImpl.java:237)
    at jetbrains.buildServer.buildTriggers.vcs.vault.connection.DelegatingVaultConnection.login(DelegatingVaultConnection.java:49)
    at jetbrains.buildServer.buildTriggers.vcs.vault.connection.ExceptionAwareConnection.login(ExceptionAwareConnection.java:115)
    at jetbrains.buildServer.buildTriggers.vcs.vault.connection.EternalVaultConnection.ensureActiveConnection(EternalVaultConnection.java:44)
    at jetbrains.buildServer.buildTriggers.vcs.vault.connection.EternalVaultConnection.getFolderVersion(EternalVaultConnection.java:90)
    at jetbrains.buildServer.buildTriggers.vcs.vault.connection.SynchronizedVaultConnection.getFolderVersion(SynchronizedVaultConnection.java:81)
    at jetbrains.buildServer.buildTriggers.vcs.vault.VaultVcsSupport.getCurrentVersion(VaultVcsSupport.java:147)
    at jetbrains.vcs.api.services.impl.RepositoryStateServiceProvider$1.getCurrentState(RepositoryStateServiceProvider.java:2)
    at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.getCurrentState(VcsRootInstanceImpl.java:186)
    at jetbrains.buildServer.buildTriggers.vcs.VcsRootChangesLoader$RunLoadChanges.run(VcsRootChangesLoader.java:80)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:74)
    at jetbrains.buildServer.buildTriggers.vcs.VcsRootChangesLoader.loadChanges(VcsRootChangesLoader.java:90)
    at jetbrains.buildServer.vcs.impl.VcsChangesFetcher$LoadChangesForRoot.run(VcsChangesFetcher.java:11)
    at jetbrains.buildServer.vcs.impl.VcsChangesLoaderImpl$ImmediateFutureExecService$1.call(VcsChangesLoaderImpl.java:3)
    at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:59)
    at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:68)
    at jetbrains.buildServer.vcs.impl.VcsChangesSyncFetcher.waitForTasksToComplete(VcsChangesSyncFetcher.java:29)
    at jetbrains.buildServer.vcs.impl.VcsChangesSyncFetcher.doLoadChanges(VcsChangesSyncFetcher.java:26)
    at jetbrains.buildServer.vcs.impl.VcsChangesSyncFetcher.tryLoadChanges(VcsChangesSyncFetcher.java:65)
    at jetbrains.buildServer.vcs.impl.VcsChangesLoaderImpl.tryLoadChanges(VcsChangesLoaderImpl.java:9)
    at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction$1.run(VcsModificationChecker.java:5)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:74)
    at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction.run(VcsModificationChecker.java:6)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon group="main" prio=5 tid=19 nid=19 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1510)
    at java.lang.Thread.run(Thread.java:745)

"OSProcessHandler pooled thread" daemon group="main" prio=10 tid=70 nid=70 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at com.intellij.execution.process.OSProcessHandler$ReadProcessThread.readNextByte(OSProcessHandler.java:305)
    at com.intellij.execution.process.OSProcessHandler$ReadProcessThread.run(OSProcessHandler.java:275)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 2" group="main" prio=5 tid=55 nid=55 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:4)
    at java.lang.Thread.run(Thread.java:745)

"Store Subversion Externals Spool Thread" daemon group="main" prio=5 tid=79 nid=79 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:634)
    at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:68)
    at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1110)

"TeamCity Diagnostics Service 1" daemon group="main" prio=5 tid=32 nid=32 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"http-nio-95-exec-9" daemon group="main" prio=5 tid=40 nid=40 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"Scheduled executor 6" group="main" prio=5 tid=178 nid=178 waiting 
   java.lang.Thread.State: WAITING

"http-nio-95-exec-6" daemon group="main" prio=5 tid=36 nid=36 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"MultiThreadedHttpConnectionManager cleanup" daemon group="main" prio=5 tid=176 nid=176 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    at mainsoft.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1101)

"TeamCity Monitor (version 10.0.4 build 42538 started 2017-04-20 15:21)" daemon group="main" prio=5 tid=18 nid=18 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.maintenance.StartupContext.waitForStageLock(StartupContext.java:146)
    at jetbrains.buildServer.maintenance.StartupContext.waitForFollowingStage(StartupContext.java:67)
    at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:493)
    at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:110)
    at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:3)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 4" group="main" prio=5 tid=57 nid=57 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:25)
    at java.lang.Thread.run(Thread.java:745)

"GC Daemon" daemon group="system" prio=2 tid=12 nid=12 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at sun.misc.GC$Daemon.run(GC.java:117)

"Normal executor 5" group="main" prio=5 tid=166 nid=166 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Store contentCache Spool Thread" daemon group="main" prio=5 tid=49 nid=49 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:634)
    at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:68)
    at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1110)

"Scheduled executor 4" group="main" prio=5 tid=51 nid=51 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"TeamCity Heartbeat" daemon group="main" prio=5 tid=38 nid=38 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:4053)
    at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226)
    at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144)
    at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099)
    at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084)
    at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427)
    at jetbrains.buildServer.serverSide.db.DBFunctions.peekValue(DBFunctions.java:1672)
    at jetbrains.buildServer.serverSide.db.DBFunctions.peekTupleValues(DBFunctions.java:1661)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuplesInternal(DBFunctions.java:1593)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuples(DBFunctions.java:1542)
    at jetbrains.buildServer.serverSide.db.HeartbeatTable.fetchAllRecords(HeartbeatTable.java:29)
    at jetbrains.buildServer.serverSide.db.Heartbeat$2.run(Heartbeat.java:704)
    at jetbrains.buildServer.serverSide.db.Heartbeat$11.run(Heartbeat.java:1138)
    at jetbrains.buildServer.serverSide.db.Heartbeat$11.run(Heartbeat.java:1135)
    at jetbrains.buildServer.serverSide.db.InternalDBUtil.runAndRetry(InternalDBUtil.java:25)
    at jetbrains.buildServer.serverSide.db.Heartbeat.runAndRetry(Heartbeat.java:1157)
    at jetbrains.buildServer.serverSide.db.Heartbeat.runAndRetry(Heartbeat.java:1135)
    at jetbrains.buildServer.serverSide.db.Heartbeat.inspectOtherRecords(Heartbeat.java:701)
    at jetbrains.buildServer.serverSide.db.Heartbeat.pulse(Heartbeat.java:963)
    at jetbrains.buildServer.serverSide.db.Heartbeat.access$200(Heartbeat.java:32)
    at jetbrains.buildServer.serverSide.db.Heartbeat$Pulser$1.run(Heartbeat.java:127)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:74)
    at jetbrains.buildServer.serverSide.db.Heartbeat$Pulser.run(Heartbeat.java:124)
    at java.lang.Thread.run(Thread.java:745)

"TC: 15:25:37 Processing watcher: File watcher for "\\nj.pfs.net\departments\TeamCity\TCData\config\build-queue-priorities.xml"; 15:25:37 Task started; Change Observer 1" daemon group="main" prio=5 tid=16 nid=16 runnable 
   java.lang.Thread.State: RUNNABLE
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.isFile(File.java:882)
    at jetbrains.buildServer.serverSide.impl.FileWatcherFactory.mainFilesExist(FileWatcherFactory.java:72)
    at jetbrains.buildServer.serverSide.impl.FileWatcherFactory.mainFilesExist(FileWatcherFactory.java:16)
    at jetbrains.buildServer.serverSide.impl.FileWatcherFactory.isWatchingEnabled(FileWatcherFactory.java:47)
    at jetbrains.buildServer.serverSide.impl.FileWatcherFactory.isWatchingEnabled(FileWatcherFactory.java:13)
    at jetbrains.buildServer.serverSide.impl.FileWatcherFactory.access$100(FileWatcherFactory.java:57)
    at jetbrains.buildServer.serverSide.impl.FileWatcherFactory$3.changesDetected(FileWatcherFactory.java:2)
    at jetbrains.buildServer.configuration.ChangeObserver.detectChanges(ChangeObserver.java:257)
    at jetbrains.buildServer.configuration.ChangeObserver.checkForModifications(ChangeObserver.java:190)
    at jetbrains.buildServer.configuration.ChangeObserver$1.run(ChangeObserver.java:227)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Normal executor 7" group="main" prio=5 tid=175 nid=175 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Connection Pool Watcher 1" daemon group="main" prio=1 tid=33 nid=33 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at jetbrains.buildServer.serverSide.db.ReliablePoolingDataSource.watchLoop(ReliablePoolingDataSource.java:124)
    at jetbrains.buildServer.serverSide.db.ReliablePoolingDataSource.access$000(ReliablePoolingDataSource.java:28)
    at jetbrains.buildServer.serverSide.db.ReliablePoolingDataSource$1.run(ReliablePoolingDataSource.java:56)
    at java.lang.Thread.run(Thread.java:745)

"Normal executor 6" group="main" prio=5 tid=174 nid=174 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 8" group="main" prio=5 tid=61 nid=61 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:25)
    at java.lang.Thread.run(Thread.java:745)

"Background Build Indexer (metadata) 1" daemon group="main" prio=1 tid=63 nid=63 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.build.index.BuildIndexer.runInBackground(BuildIndexer.java:89)
    at jetbrains.buildServer.serverSide.build.index.BuildIndexer.access$000(BuildIndexer.java:59)
    at jetbrains.buildServer.serverSide.build.index.BuildIndexer$2.run(BuildIndexer.java:3)
    at java.lang.Thread.run(Thread.java:745)

"Usage statistics collector 1" daemon group="main" prio=5 tid=77 nid=77 waiting 
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at jetbrains.buildServer.usageStatistics.impl.UsageStatisticsCollectorImpl.waitForEvent(UsageStatisticsCollectorImpl.java:210)
    at jetbrains.buildServer.usageStatistics.impl.UsageStatisticsCollectorImpl.run(UsageStatisticsCollectorImpl.java:136)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 7" group="main" prio=5 tid=60 nid=60 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:4)
    at java.lang.Thread.run(Thread.java:745)

"HSQLDB Timer @db370b" daemon group="main" prio=5 tid=62 nid=62 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at org.hsqldb.lib.HsqlTimer$TaskQueue.park(HsqlTimer.java:882)
    at org.hsqldb.lib.HsqlTimer.nextTask(HsqlTimer.java:528)
    at org.hsqldb.lib.HsqlTimer$TaskRunner.run(HsqlTimer.java:610)
    at java.lang.Thread.run(Thread.java:745)

"main" group="main" prio=5 tid=1 nid=1 runnable 
   java.lang.Thread.State: RUNNABLE
    at java.net.DualStackPlainSocketImpl.accept0(Native Method)
    at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:452)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:781)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:727)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:428)

"OSProcessHandler pooled thread" daemon group="main" prio=5 tid=69 nid=69 runnable 
   java.lang.Thread.State: RUNNABLE
    at java.lang.ProcessImpl.waitForInterruptibly(Native Method)
    at java.lang.ProcessImpl.waitFor(ProcessImpl.java:449)
    at com.intellij.execution.process.OSProcessHandler$ProcessWaitFor$1.run(OSProcessHandler.java:97)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Store issues Spool Thread" daemon group="main" prio=5 tid=66 nid=66 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:634)
    at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:68)
    at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1110)

"Agents Ping Executor 1" daemon group="main" prio=5 tid=177 nid=177 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Memory Usage Watcher 1" daemon group="main" prio=5 tid=64 nid=64 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at jetbrains.buildServer.diagnostic.MemoryUsageMonitor$Watcher.run(MemoryUsageMonitor.java:69)
    at java.lang.Thread.run(Thread.java:745)

"Flush Queue 1" daemon group="main" prio=5 tid=170 nid=170 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at jetbrains.buildServer.a.m.b(m.java:98)
    at jetbrains.buildServer.a.m.a(m.java:103)
    at jetbrains.buildServer.a.m.hasEnterpriseLicense(m.java:109)
    at jetbrains.buildServer.a.m.isUnlimitedBuildTypes(m.java:80)
    at jetbrains.buildServer.a.j.isMaxNumberOfBuildTypesExceeded(j.java:13)
    at jetbrains.buildServer.serverSide.impl.BuildServerImpl.flushQueue(BuildServerImpl.java:56)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$2.doSomething(BuildServerRunner.java:2)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.runAction(BuildServerRunner.java:43)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:52)
    at java.lang.Thread.run(Thread.java:745)

"ObsoleteBuildsKiller 1" group="main" prio=5 tid=162 nid=162 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Low prio executor 4" daemon group="main" prio=4 tid=160 nid=160 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Periodic tasks thread" daemon group="main" prio=5 tid=73 nid=73 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"http-nio-95-exec-3" daemon group="main" prio=5 tid=26 nid=26 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"ImmutableAgentInfo-cleanup 1" daemon group="main" prio=5 tid=168 nid=168 waiting 
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    at jetbrains.buildServer.serverSide.flakyTestDetector.opendata.ImmutableAgentInfo.lambda$startCleanup$1(ImmutableAgentInfo.java:263)
    at jetbrains.buildServer.serverSide.flakyTestDetector.opendata.ImmutableAgentInfo$$Lambda$161/30453002.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)

"Vcs Monitor 1" daemon group="main" prio=5 tid=171 nid=171 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
    at java.lang.StringBuilder.append(StringBuilder.java:136)
    at jetbrains.buildServer.vcs.VcsUtil.propertiesToString(VcsUtil.java:200)
    at jetbrains.buildServer.vcs.impl.VcsRootInstancesManagerImpl.propertiesHash(VcsRootInstancesManagerImpl.java:27)
    at jetbrains.buildServer.vcs.impl.VcsRootInstancesManagerImpl.makeId(VcsRootInstancesManagerImpl.java:25)
    at jetbrains.buildServer.vcs.impl.VcsRootInstancesManagerImpl.findOrCreateVcsRootInstance(VcsRootInstancesManagerImpl.java:100)
    at jetbrains.buildServer.vcs.impl.VcsRootInstancesManagerImpl.getVcsRootInstance(VcsRootInstancesManagerImpl.java:187)
    at jetbrains.buildServer.vcs.impl.VcsRootInstancesManagerImpl.getVcsRootInstance(VcsRootInstancesManagerImpl.java:177)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.resolveEntries(BuildTypeImpl.java:1022)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.getVcsRootInstanceEntries(BuildTypeImpl.java:94)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.getVcsRootInstanceEntries(BuildTypeImpl.java:523)
    at jetbrains.buildServer.serverSide.impl.VcsModificationChecker.checkForModificationsAsync(VcsModificationChecker.java:194)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$3.doSomething(BuildServerRunner.java:2)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.runAction(BuildServerRunner.java:43)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:52)
    at java.lang.Thread.run(Thread.java:745)

"Low prio executor 3" daemon group="main" prio=4 tid=159 nid=159 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"RMI Scheduler(0)" daemon group="system" prio=5 tid=75 nid=75 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"http-nio-95-exec-2" daemon group="main" prio=5 tid=25 nid=25 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"Normal executor 3" group="main" prio=5 tid=156 nid=156 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"http-nio-95-exec-5" daemon group="main" prio=5 tid=35 nid=35 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"Scheduled executor 5" group="main" prio=5 tid=52 nid=52 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 6" group="main" prio=5 tid=59 nid=59 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:4)
    at java.lang.Thread.run(Thread.java:745)

"TC: 15:25:17 scan for agent plugins packages; 15:25:17 Task started; Agent Distribution Monitor 1" group="main" prio=5 tid=163 nid=163 runnable 
   java.lang.Thread.State: RUNNABLE
    at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)
    at java.io.File.isDirectory(File.java:849)
    at jetbrains.buildServer.serverSide.impl.agent.AgentPluginsProviderBase.getPlugins(AgentPluginsProviderBase.java:1)
    at jetbrains.buildServer.serverSide.impl.agent.CompositeAgentPluginsHolder.getPlugins(CompositeAgentPluginsHolder.java:25)
    at jetbrains.buildServer.serverSide.impl.agent.AgentPluginsVersionProvider.getUpdateDirTimestamp(AgentPluginsVersionProvider.java:41)
    at jetbrains.buildServer.serverSide.impl.agent.AgentPluginsVersionProvider.updateVersion(AgentPluginsVersionProvider.java:50)
    at jetbrains.buildServer.serverSide.impl.agent.AgentPluginsVersionProvider.getPluginsVersion(AgentPluginsVersionProvider.java:34)
    at jetbrains.buildServer.serverSide.impl.agent.AgentUpdateInfoProviderImpl.updateVersion(AgentUpdateInfoProviderImpl.java:54)
    at jetbrains.buildServer.serverSide.impl.agent.AgentUpdateInfoProviderImpl$1.run(AgentUpdateInfoProviderImpl.java:46)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:74)
    at jetbrains.buildServer.util.ExceptionUtil$1.run(ExceptionUtil.java:41)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Normal executor 1" group="main" prio=5 tid=154 nid=154 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Finalizer" daemon group="system" prio=8 tid=3 nid=3 waiting 
   java.lang.Thread.State: WAITING
    at java.lang.Object.wait(Native Method)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"NuGet Packages Version Checker 2" daemon group="main" prio=5 tid=179 nid=179 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 3" group="main" prio=5 tid=56 nid=56 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:4)
    at java.lang.Thread.run(Thread.java:745)

"NuGet Packages Version Checker 1" daemon group="main" prio=5 tid=167 nid=167 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"http-nio-95-ClientPoller-0" daemon group="main" prio=5 tid=20 nid=20 runnable 
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1198)
    at java.lang.Thread.run(Thread.java:745)

"Low prio executor 2" daemon group="main" prio=4 tid=158 nid=158 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"TC: 15:25:33 Save File On Disk; BackgroundPersister - Save 1" group="main" prio=5 tid=47 nid=47 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
    at jetbrains.buildServer.serverSide.impl.projects.BackgroundPersisterImpl.awaitCompleteTask(BackgroundPersisterImpl.java:96)
    at jetbrains.buildServer.serverSide.impl.projects.BackgroundPersisterImpl.access$100(BackgroundPersisterImpl.java:199)
    at jetbrains.buildServer.serverSide.impl.projects.BackgroundPersisterImpl$FlushOnDisk.run(BackgroundPersisterImpl.java:38)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:74)
    at jetbrains.buildServer.util.ExceptionUtil$1.run(ExceptionUtil.java:41)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Alarm (Queue pause/resume watcher)" daemon group="main" prio=5 tid=78 nid=78 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"OSProcessHandler pooled thread" daemon group="main" prio=5 tid=72 nid=72 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at com.intellij.util.concurrency.Semaphore.waitFor(Semaphore.java:60)
    at com.intellij.execution.process.OSProcessHandler$ProcessWaitFor.waitFor(OSProcessHandler.java:109)
    at com.intellij.execution.process.OSProcessHandler$3$1.run(OSProcessHandler.java:144)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Low prio executor 5" daemon group="main" prio=4 tid=164 nid=164 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"DBMaintenanceScheduler" daemon group="main" prio=5 tid=149 nid=149 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

"Flush Web UI Updates 1" daemon group="main" prio=5 tid=65 nid=65 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at jetbrains.buildServer.push.impl.DefaultTopicsRegistry$ValueTopicsFlushProcessor.flush(DefaultTopicsRegistry.java:22)
    at jetbrains.buildServer.push.impl.DefaultTopicsRegistry$ValueTopicsFlushProcessor.lambda$start$0(DefaultTopicsRegistry.java:3)
    at jetbrains.buildServer.push.impl.DefaultTopicsRegistry$ValueTopicsFlushProcessor$$Lambda$25/28942224.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 5" group="main" prio=5 tid=58 nid=58 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:4)
    at java.lang.Thread.run(Thread.java:745)

"Scheduled executor 3" group="main" prio=5 tid=50 nid=50 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Long.valueOf(Long.java:840)
    at com.sun.proxy.$Proxy23.schedule(Unknown Source)
    at jetbrains.buildServer.serverSide.RunBuildsStatusManager.getBuilds(RunBuildsStatusManager.java:24)
    at jetbrains.buildServer.push.notificator.RunningBuildsNotificator$1.run(RunningBuildsNotificator.java:5)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Agent messages processing thread 1" group="main" prio=5 tid=54 nid=54 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue.getLockedNotEmptyQueue(BuildAgentMessagesQueue.java:34)
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue.access$300(BuildAgentMessagesQueue.java:73)
    at jetbrains.buildServer.serverSide.impl.BuildAgentMessagesQueue$2.run(BuildAgentMessagesQueue.java:18)
    at java.lang.Thread.run(Thread.java:745)

"net.sf.ehcache.CacheManager@1dc88fa" daemon group="main" prio=5 tid=42 nid=42 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    at java.util.TimerThread.run(Timer.java:505)

"http-nio-95-exec-8" daemon group="main" prio=5 tid=39 nid=39 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"Low prio executor 1" daemon group="main" prio=4 tid=157 nid=157 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Thread-3" daemon group="main" prio=5 tid=17 nid=17 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:103)

"TC: 15:25:37 Ping 1" daemon group="main" prio=5 tid=169 nid=169 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:8)
    at java.lang.Thread.run(Thread.java:745)

"Alarm pool(shared)" daemon group="main" prio=3 tid=74 nid=74 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Scheduled executor 2" group="main" prio=5 tid=45 nid=45 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"Store VcsRootInstanceData Spool Thread" daemon group="main" prio=5 tid=43 nid=43 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:634)
    at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:68)
    at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1110)

"Scheduled executor 1" prio=5 tid=44 nid=44 
    at java.lang.Throwable.printStackTrace(Throwable.java:643)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1061)
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
    at java.lang.Thread.dispatchUncaughtException(Thread.java:1952)

"http-nio-95-exec-10" daemon group="main" prio=5 tid=41 nid=41 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"NioBlockingSelector.BlockPoller-1" daemon group="main" prio=5 tid=13 nid=13 runnable 
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:339)

"http-nio-95-exec-4" daemon group="main" prio=5 tid=34 nid=34 waiting 
   java.lang.Thread.State: WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
    at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

"Store issue-errors Spool Thread" daemon group="main" prio=5 tid=67 nid=67 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:634)
    at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:68)
    at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1110)

"http-nio-95-ClientPoller-1" daemon group="main" prio=5 tid=21 nid=21 runnable 
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
    at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
    at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1198)
    at java.lang.Thread.run(Thread.java:745)

"Background Build Indexer (search) 1" daemon group="main" prio=1 tid=146 nid=146 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Object.wait(Native Method)
    at jetbrains.buildServer.serverSide.build.index.BuildIndexer.runInBackground(BuildIndexer.java:89)
    at jetbrains.buildServer.serverSide.build.index.BuildIndexer.access$000(BuildIndexer.java:59)
    at jetbrains.buildServer.serverSide.build.index.BuildIndexer$2.run(BuildIndexer.java:3)
    at java.lang.Thread.run(Thread.java:745)

"Test failure rate persisting thread 1" daemon group="main" prio=5 tid=53 nid=53 waiting 
   java.lang.Thread.State: TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at jetbrains.buildServer.serverSide.stat.TestFailureRateCollector.processPersistingQueue(TestFailureRateCollector.java:26)
    at jetbrains.buildServer.serverSide.stat.TestFailureRateCollector.access$500(TestFailureRateCollector.java:69)
    at jetbrains.buildServer.serverSide.stat.TestFailureRateCollector$3.run(TestFailureRateCollector.java:1)
    at java.lang.Thread.run(Thread.java:745)

"Attach Listener" daemon group="system" prio=5 tid=4 nid=4 runnable 
   java.lang.Thread.State: RUNNABLE


Memory usage:
   Code Cache: used = 45.24 MB committed = 45.9 MB max = 240 MB used/max = 18.85%
   Metaspace: used = 101.73 MB committed = 104.25 MB max = -1 B 
   PS Eden Space: used = 20.19 MB committed = 57 MB max = 57 MB used/max = 35.43%
   PS Survivor Space: used = 0 B committed = 56.75 MB max = 56.75 MB used/max = 0.00%
   PS Old Gen: used = 321.11 MB committed = 341.5 MB max = 341.5 MB used/max = 94.03%
   Total: used = 488.28 MB max = 695.24 MB used/max = 70.23%

CPU statistics:
   Available processors = 8
   Java process = 95.89%
Full GC started 2ms ago, took 411ms, 66.9 KB freed
In previous 1 minute full GC was fired 111 times, consuming 50s,335ms and freeing 593.38 MB in total

Dump taken in 13ms
Please sign in to leave a comment.