How to customise maximum VCS queue capacity

Hi Peter,

How many VCS Roots do you have? What's their changes checking interval like? Do they access different servers, different resources or do you have duplicated ones? Are you hitting this limit consistently? Are you using any third party plugin for VCS roots? In a large enough installation, it might make sense to increase the limits, but it would be useful to have the information beforehand.

If your installation is large enough, and you don't want to share this information, we have the internal properties:
teamcity.threadpool.vcsexecutor.threads.max=50
teamcity.threadpool.vcsexecutor.queue.capacity=5000

It's probable you can use this two to increase your VCS threads and queue capacity, but consider reviewing whether you can improve your VCS Configurations and check whether any of the VCS plugins is misbehaving.

Hi Denis,

Our VCS root count is currently 893 (all unique). Of those, 1 is a GitHub Enterprise server, and the remaining are an SVN server; no third-party plug-ins are installed. They all use a global minimum checking interval of 60 seconds.

This is more a case of our QA team misbehaving ;). We have automated tests with snapshot dependencies on our source code (which itself is a long chain of dependencies). Because of the way TeamCity works, each time one of the automated test projects is added to the build queue, it initially adds a chain of about 30 projects. Usually, TeamCity then figures out that 29 of these don't need building, due to re-using existing builds, and so the queue is trimmed back down to 1 project. Unfortunately, each Friday night our QA team use a script to add approximately 100 automated test projects into the build queue (via the REST API). This results in over 3000 projects being added, which hit both the build queue limit (which I then increased via teamcity.buildTriggersChecker.queueSizeLimit), and this VCS executor queue limit. The only way I was then able to recover the server was by clearing the queue, and then restarting the TeamCity server (clearing the queue did not resolve the problem by itself). This test project count has steadily increased over time, and so presumably in the past couple of weeks we have finally hit this ceiling.

As a work-around, I've advised our QA team to throttle their script, so that TeamCity has the chance to remove redundant builds before the queue size hits these limits. I'd also like to increase the VCS executor queue limit, just as a precaution, so the server doesn't get wiped out again.

For reference, we're using TeamCity Enterprise 2017.2.1 (build 50732), with Java version: 1.8.0_151 (x64). These are our JVM arguments:

-Djava.util.logging.config.file=c:\TeamCity\bin\..\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Xrs -Xmx8g -XX:MaxPermSize=270m -XX:ReservedCodeCacheSize=350m -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j.configuration=file:../conf/teamcity-server-log4j.xml -Dteamcity_logs=c:\TeamCity\bin\..\logs -Dcatalina.base=c:\TeamCity\bin\.. -Dcatalina.home=c:\TeamCity\bin\.. -Djava.io.tmpdir=c:\TeamCity\bin\..\temp 

Thanks for your help!

Hi Peter,

well, that's certainly a way to keep servers busy for the weekend. I would definitely say that this is a good scenario to use the mentioned parameters, those should give you (and QA) more room to wiggle. If you keep running into issues don't hesitate to report back, we'll see if we can improve the configuration for that specific scenario

Hi Denis,

I added the above lines to the internal properties and then restarted the server, however the same error is still showing in the server log.

My exact steps were:

1. I added the settings via the Diagnostics page in the web interface;
2. At this stage, I didn't restart the server;
3. Over the weekend the build queue got to nearly 3000 items again;
4. I looked in this logs this morning, and noticed the same error;
5. I figured that maybe a server restart is needed, and so restarted the server _without_ removing the queued items;
6. After the restart, I noticed the same error in the logs.

Am I missing anything else?

To confirm, here's the error again:

[2018-03-26 09:12:44,184]   WARN [ges collector 5] -      jetbrains.buildServer.VCS - Collecting changes for build configurations [...] failed: java.util.concurrent.RejectedExecutionException: Unable to put new task into the queue because maximum capacity is exceeded. Current queue size: 1000. Executor info: Thread factory "VCS Generic executor" {corePoolSize=3; maxPoolSize=30; activeCount=30} (enable debug to see stacktrace) 

And here are the internal properties:

Thanks for any assistance!

Hi,

  Correct pool settings for this executor are:

  teamcity.threadpool.base.vcsexecutor.queue.capacity=5000
  teamcity.threadpool.base.vcsexecutor.threads.max=50
  teamcity.threadpool.base.vcsexecutor.threads.core=20

 Hope this helps,

These settings worked perfectly, thanks! No further issues have been seen with stuck builds in the queue, since applying the latest VCS executor settings (and restarting the server, for good measure).

Hi Kirill,

I see in most forums and in youtrack that the property to set large number of threads is

teamcity.threadpool.vcsexecutor.threads.max=20

However what you suggested above is somewhat different i.e. contains base in property name.

teamcity.threadpool.base.vcsexecutor.threads.max=50

If using 2017.2.3 which property is the correct one to use? Is there any official documentation on your confluence page regarding all available internal properties supported by TeamCity?

Hi Richard,

The name of the property depends on the problem you have. teamcity.threadpool.vcsexecutor.threads.max is used to set number of threads which periodically poll VCS repositories for changes, i.e. when you have many VCS roots with small checking for changes interval and you want to changes to appear faster. The alternative approach in this case would also be using post-commit hooks: https://confluence.jetbrains.com/display/TCDL/Configuring+VCS+Post-Commit+Hooks+for+TeamCity

Property teamcity.threadpool.base.vcsexecutor.threads.max is used when collecting changes in large build chains when they are either in the build queue or starting. Basically, it should be increased when you have problems with "VCS Generic executor", as in the original message.

We don't have official docs regarding internal properties, as there are hundreds of them, and we'd like to know when and how they are used (and to avoid bugreports caused by their incorrect usage).

Hope this helps,

Hi Kirill,

Thanks, that clarifies things nicely. 

In our case we have a large number of VCSRoots (~17k) and we enforce server default polling period of 2 hours to encourage teams to use Commit Hooks like so

http://teamcity/guestAuth/app/rest/vcs-root-instances/commitHookNotification?locator=vcsRoot:(type:jetbrains.git,property(name:url,value:${project.key}/${repository.slug}.git,matchType:contains,ignoreCase:true)) 

It is my understanding that this would essentially trigger a check for changes action from TeamCity to the Git Server (BitBucket). If that is the case would setting teamcity.threadpool.vcsexecutor.threads.max then still apply in this case?

Thanks in advance.

Hi Richard,\

  With your setup I'd expect you don't need to increase any properties. Why would you want to increase any properties?

  Regards,

Hi Kirill,

1. Currently only ~200 VCSRoots have commit hooks configured so the vast majority are still polling. We have cases where some even when users manually force checking for changes it takes a long time for the operation to complete as there are so many in the queue and some are long running (a separate issue).

2. If commit hooks were configured for all VCSRoots and a large number were triggered within in a short time frame would this not essentially replicate the same problem? i.e. large number of checking for changes operations in a queue?

Thanks,

Richard

Hi Richard,

   Sorry for delay with the answer.

   Both commit hooks and manual checking for changes operations mostly use `teamcity.threadpool.vcsexecutor.threads.max` pool, so I suppose it is the pool you'd like to increase. Please make sure you have enough CPU cores, as increasing this pool size increase CPU load on the server (you can see the CPU load on the diagnostics page of admin UI).

   In the case you're describing, with many hooks starting checking for changes, the larger size of the pool may help as well. 

   Hope this helps,

Can you please explain the difference between the 2 properties ? 
teamcity.threadpool.vcsexecutor.threads.max
teamcity.threadpool.base.vcsexecutor.threads.max

I have a 36 core server and would like to bring down the time it takes to check for VCS changes. 
Our polling period is set to 10 hours. 
My goal is to decrease the time to check for changes in VCS roots when builds are kicked off via rest api. 

Hello Paul,

   Before playing with executors, you should probably investigate what is the real problem in your setup.

   This VCS status page may help to figure out the current status and find out which VCS roots are slow: http://your-server/admin/admin.html?item=diagnostics&tab=vcsStatus

   Thread dumps from server during the checking for changes phase is the another source of information.

   But in general, if all your VCS roots use commit hooks for triggering, and these hooks are granular enough (i.e. commit hooks trigger only relevant builds), there should not be a problem with executor pools.

  Kind regards,