TeamCity server sometimes "forgets" which branches have already been built, leading to build flood
On different occasions, we have encountered the situation where a given TeamCity config that is set-up with a VCS trigger will simply "lose" its reference point as to which branches have already been built for a given VCS root. This will usually happen after a TeamCity upgrade, or sometimes simply after a service restart. The only VCS system which we've seen this happen on is Git (but then it's the only VCS system we have that is using branches).
When this happens, the VCS trigger will simply fire off a build for every branch that exists in the repository (and that is tracked by the VCS root). In our case, this means that about 3000+ builds are queued, which we then have to painstakingly cancel, after making sure the config(s) is paused or archived and does not keep on triggering builds of course. Even worse, un-pausing the configuration will usually make this flood of builds start again. The only way we've found to make sure that the configuration does not get back into this state is to disable the VCS triggers before un-pausing the configuration.
Is this a known issue? How can this be avoided? There is very little information available as to "how" the state of a branch is persisted on the TeamCity side, and what decides if a branch's latest changes should be built or not. For example, as far as I know, adding a VCS trigger to an existing repository with various different branches will NOT automatically trigger a build for each of these branches, but adding a new commit after the VCS trigger in added/enabled will. Does this mean that TeamCity captures a "snapshot" of the state of the changes when a VCS trigger is added/enabled? If that's the case, how can this lead to the situation we're witnessing? What happens in the case of multiple VCS roots?
Please sign in to leave a comment.
The way I see this is that teamcity relies on the commit hash vor VCS changes (at least that is logged in the builds).
But TeamCity also tries to have repeatable builds. Which means that configuration changes will also count as change. They will not fire a trigger, but should the build be part of some build chain, it will be re-run.
So: are these builds part of a build chain (having Snapshot dependencies or are used as one)? Or do you pass parameters up the build chain (using those "reverse.dep.*.paramName" parameters)?
We had a similiar effect that builds would run too often or for no obvious reason. Our cause was the build configuration changed. We had one build chain with these reverse.dep. Parameters, and another one with different settings. Both chains would run one after the other. But teamcity detected this as changed configuration, and to be sure it re-ran a lot of builds (since it cant know this paramter is only used to get passed to the deployment tool and does not affect the builds themselves).
So a changed config (or a suspected change) might be the reason. You could have a look at the parameters of two builds if they differ. You can also go the the artifacts tab of a build, show the hidden artifacts and compare the buildSettings.xml file. That might give you a hint on what teamcity thinks is going on.
Hi,
Werner's comment is pretty much correct. TeamCity tracks commit hashes (we call them revisions to generalize for all different kinds of VCS Roots) and will trigger on new changes, but also on some configuration changes. This said, the initial comment seems like it just triggers on a server restart. This shouldn't happen in general, and might be an issue under some specific environment, so if you could recreate a project where it happens consistently and share it with us (even privately) via our issue tracker, that would help us replicating the problem and locating any possible issue in it.