Teamcity not respecting branches in dependent builds

We have a build pipeline consisting of build steps, deploy steps, integration test projects. All these projects are set up in a pipeline, with snapshot dependencies. They are all connected to the same VCS root, a git repo, which has branch support enabled.

Usually the builds work fine, and the builds use the dependencies from the correct branch, but quite often (maybe when multiple branches are building simultaneously, but I'm not certain), the builds further down in the pipeline use snapshot dependencies from the wrong branch. It even triggers a new build of the dependency with default branch, not the branch/changeset selected.

I have even experienced this in newly developed build pipelines, only operating on a specific branch; suddenly a build of the default branch is triggered. Please see attached screenshot.

 

13 comments
Comment actions Permalink

Hello Erik,

Could you please also attach screenshot of the VCS root settings and teamcity-server.log file? What triggers do you have configured? 

0
Comment actions Permalink

I'm not very keen on uploading the teamcity-server.log file in its entirety. It may contain information we do not want public. Are there any special parts you are looking for? 

There are configured some "Finish build" triggers, but mostly we use snapshot dependencies, and trigger the "downmost" project in the pipeline.

0
Comment actions Permalink

Hello Erik, 

Do you have %teamcity.build.branch% redefined in any build configuration?
From the server log we will understand how the builds were triggered. Could you please use "Submit a request" button on the top to send us logs privately?

0
Comment actions Permalink

No redefinition of %teamcity.build.branch% anywehre. I will send logs privately, thanks.

0
Comment actions Permalink

Please see support request 803508

0
Comment actions Permalink

Hi Erik,

What TeamCity version do you use?

Can it be that the revision the builds run on is included into the default branch as well?

0
Comment actions Permalink

TeamCity can take build from the default branch if this build is suitable by revisions.


For instance, there is a build in default branch on some revision sha1, and chain in some non-default branch is added into the queue. At this point TeamCity will collect changes and determine revisions for all of the builds in the chain. If revision of some of the queued build in non-default branch is the same as revision of the build in default branch (same configuration), then TeamCity can remove queued build and substitute it with the build from default branch, because this replacement does not change end result. This replacement works to default branch only, TeamCity never replaces default branch with build from some other branch.

So it's not like TeamCity mixes branches, it's more like revision is just more important. If you find this problematic, please provide details why.

0
Comment actions Permalink

Hi.

This could make sense, except it's not the same revisions being build in default branch and the feature branch, and, since Teamcity sometimes doesn't check out the right revision (see https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206227519-TeamCity-is-checking-out-the-wrong-VCS-revision), we have stopped using checkout rules, so it's not because of that.

The "develop" (default branch) builds a totally different revision, which hadn't been merged into the feature branch at the time. So this explanation does not explain the issue.

 

That said, begin "smart" on choosing which branch to build, is not a goog strategy for our use of Teamcity, as we use the git chechout being build to set version number on the build (in a step we depend on, which then gets the wrong version number too), and we use it to set environment names, as we deploy branches to multiple test environments, etc.

 

If there is a flag to tell TeamCity to "stop being smart", and just building the revision we tell it to build, it would make it a much more usable product. We have so many issues now because of TeamCity trying to be smarter than git when deciding what to build.  It's closing in on not usable for our day-to-day business now, I'm afraid.

 

0
Comment actions Permalink

This thread: https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206227519-TeamCity-is-checking-out-the-wrong-VCS-revision is pretty old and is about TeamCity 6.5. I don't think it is relevant.

I'd still stick with my explanation. Note that these are two separate processes - one process fixes revisions in queued builds, and another one - optimizes builds in queue based on these revisions. For some reason the first process set the same revisions as in default branch to builds in configurations: Public Api, Api, web and generate release log. You said this is incorrect, then let's try to find out why it happened.

Are all of the build configurations have the same branch specification? Is it possible that problematic build configurations use VCS roots without branch specification?

0
Comment actions Permalink

The thread is updated in January, and we still have this problem. We have reported it here, but never really got to the bottom of it: https://youtrack.jetbrains.com/issue/TW-46810. But, however, this is a separate issue.

 

All builds have the same VCS root attached. Same VCS branch setup. Did you look at the logs I uploaded? Did they tell you anything?

0
Comment actions Permalink

Date in this thread is incorrect. We used other forums for several years, and migrated to ZenDesk forums recently. Unfortunately thread date was not preserved during migration. Note that thread is about TeamCity 6.5 which was released on 24th of May, 2011. 

 

As to logs, unfortunately I could not recreate the whole scenario from them. They contain useful information but lack details about your configuration.

Could you please clarify the purpose of "Set version number" build configuration? From the name it looks like it provides build number to all other build configurations in chain, is this correct? Does it have VCS roots? 

0
Comment actions Permalink

Hello. The problem was solved?

Teamcity 10.0.5 (build 42677).

Big build chain. 30+ project. 1 or more VCS (git only) on project. All VCS are configured the same way. Differed only default branch. VCS trigger in last build: all branch and "Trigger a build on changes in snapshot dependencies" enabled. Snapshot depends in all build. Artifact depends "Build from the same chain".

Teamcity select default branch if build with default branch completed after selected branch.

because of this I get broken builds. I am looking for any way to fix this.

We noticed the problem only now. The support period has expired. I will try to make a minimal example reproducing the problem. But can there already be a solution?

0
Comment actions Permalink

Hello Александр,

It makes sense to start a new thread as this one is too old and your issue is most probbaly not directly related.

BTW, It is highly recommended to upgrade to the latest TeamCity (2018.1.2 as of now): not only you get a supported version with lots of fixes and new features, but you also get the latest security patches and do not continue to use a version with known vulnerabilities.

0

Please sign in to leave a comment.