Misbehaving VCS Triggers
We are evaluating TeamCity to see how well it can integrate with our Gerrit instance. Unfortunately, we ran into what appears to be some sort of scaling issue with VCS triggers. We've added around 25 VCS roots and had TeamCity configured to trigger builds on changes in refs/changes, but it stopped working as expected as soon as we added another VCS root that has large number of open change lists on our Gerrit server.
It looks like TeamCity is picking new changes, but it's not triggering the builds as expected. Oddly enough, builds either never get triggered or get triggered with 15 mins to 1 hour delay, and this is affecting all VCS roots on our installation.
After going through debug logs I am thinking that large number of branches in refs/changes/* is causing problem(s) with VCS triggers. Here are some of the messages I see in teamcity-server.log:
teamcity-server.log:[2020-03-06 17:53:52,252] INFO - Side.impl.BuildTriggersChecker - Long trigger execution time of 20m:23s,508ms for TriggerDescriptor{id='vcsTrigger', name='vcsTrigger'} of experimental / presubmit/internal {id=experimental_presubmit_internal, internal id=bt43}
teamcity-server.log:[2020-03-08 00:36:33,497] WARN - Side.impl.BuildTriggersChecker - Long all triggers execution time of 34m:50s,038ms: there are misbehaving triggers, build queueing can be delayed
It looks as VCS trigger mechanism is taking long time to process VCS Root with large number of open change lists. Is there anything we can do about it? We have hundreds VCS Roots and some do have a lot of changes.
Please sign in to leave a comment.
Just a follow up, it appears as if triggering mechanism started working as expected as soon as I changed branch spec in VCS root from +:refs/(changes/*) to +:refs/(changes/52/371952/*). Limiting branch spec to just one change list did the trick, which makes it more likely that large number change lists in refs/changes namespace is slowing down TeamCity's triggering mechanism.
Any ideas on a workaround?
Hi Igor, sorry for the delay.
When processing triggers TeamCity has to go through the changes on all the tracked branches. The speed at which this happens will obviously depend on the server power, but based on it, will increase as the number of branches grows. This is especially true for initial runs as caches have to get built and the data is new.
I am not particularly aware of special issues with gerrit and number of changes, we are used to large numbers of branches and those don't normally produce such huge delays. While a combination of factors might lead to such long triggering, if the behaviour continues, we would recommend taking some thread dumps while it happens as described here: https://www.jetbrains.com/help/teamcity/reporting-issues.html#ReportingIssues-Slowness,HangingsandLowPerformance, then forwarding those to us for review, when possible sharing with us an estimate of the number of branches that we are talking about.
The workaround for this kind of scenario is trying to reduce the number of branches tracked to those appropriate, rather than all those that exist and match a wildcard. This said, it would make sense if we could go over the investigation described above as this process taking more than a few seconds is already rare.