TeamCity do not detect changes in Git branches when merging / rebasing

Background:

We have the following set up with TeamCity Enterprise 8.1.1:

  • One git repository with three branches:
    • develop
    • master
    • release
  • One TC project linked to a single VCS root:
    • Default branch: refs/heads/develop
    • Branch specifciation: +:refs/heads/*


In my TC project, I have a VCS trigger (default options, no filters)

If I make a change and push to develop, master or release, TC detects a change in develop and builds it just as expected.

Problem:

  1. I make a change to the develop branch
  2. I push that change
    A build now starts for the develop branch. While this build is running:
  3. I run git checkout master
  4. I run git rebase develop
  5. I run git push
    At this point both develop and master points at the same change set in git.

At this point, I would expect a build to start on the master branch as well, but that does not happen. The build on the develop branch finishes a few minutes later, but nothing is built for master branch.

Some problem isolation:

  • When I run the above steps, after step 5 TeamCity does not show any pending changes for the master branch. I expect changes to be shown since I just merged the develop branch into master and pushed.
  • I've tried running git merge develop instead of git rebase develop in step 4 but that does not help.
  • If I make my change locally both in master and in develop and then push both branches at the same time, then both branches are built.
  • If TeamCity is busy and the build is queued in step 2 and I push while it's queued, then a build for both branches will be queued.
  • If TeamCity starts to build the develop branch in step 2 and I stop this, then the master branch is built after step 5.
24 comments
Comment actions Permalink

I have the same problem. If use release branches in git, and if I fast-forward merge to master (which has already been built) TeamCity doesn't trigger a build, and if I manually build it says there are no changes.

 

 

2
Comment actions Permalink

The issue is that I have 2 git branches: master and a release branch. When the release branch and master branch point to the same changeset after a merge (i.e. the release branch was fast forwarded), TC does not detect changes on the release branch. I set the property teamcity.vcsTrigger.runBuildOnSameRevisionInEveryBranch to "true" but it appears to not have an effect. The only way to get TC to recognize changes on the release branch is to add another commit to just that branch.

1
Comment actions Permalink

I'm facing similiar issues pushing different branches. Our workflow are basically the same described by Matthew. We have two branches, master and release. Master being the default branch. Development is made in master, and when we are satisfied with the changes, we do a merge/PR to branch release. We are currently using TeamCity 2017.2.2 (build 50909) and GitLab for VCS.

When we do some changes in release and then push, TeamCity builds correctly. Then we do a fast-foward to master and then push. TeamCity detects pending changes on master and then builds.

The problem occurs when branch master is pushed first. For example. We do some changes in master and push. TeamCity detects pending changes and then build. Next we fast-foward master to release and then push release. TeamCity does not find any pending changes in release.

Looking for help I found this information, which says:

  • If the default branch is one of the branches in the merging/fast-forwarding, the changes are always calculated against the default branch, if there is a build on same revision in the default branch, TeamCity will not run a new build on the same revision.

For my understanding heres is where it lies my problem. Since master (default branch) was pushed first and release last changes came from fast-fowarding master, TeamCity will calculate the changes against master, the default branch, and not against release, the branch being pushed.

1
Comment actions Permalink

Similar issue here with v2019.1.2, although in our case the builds are triggering but the changes are not being tracked and reported correctly.

We have a develop branch, and periodically we'll do a fast-forward merge from develop to a QA branch (these merges only ever happen one-way).

Any PR that commits to develop first show up as "Pending" on the build config and then triggers a build in short order. While building, the changes are still visible on the dashboard and after building the changes reported for that build are correct.

Whenever we merge from develop -> QA, we first see the "Pending" changes for that branch on the build config, and the build is properly triggered. However, as soon as the build starts, the dashboard shows "No changes". After the build finishes, the dashboard still shows "No changes", as does the "Changes" tab for the build. What's stranger is that the "Changes" tab after the build does actually show a link to a "revision" (commit hash?) which when clicked shows the merge change details (files, comment and user).

1
Comment actions Permalink

How the hell is this still an issue, Im facing the same issue right now..

1
Comment actions Permalink

I'm experiencing the same issue, is there any fix on this? I would expect a fast-forward rebase or merge to a branch to trigger a build regardless of whether that changeset have been built on a different branch.

0
Comment actions Permalink

I'm running 10.0.4 and have the same issue. When 2 branches point to the same changeset, one of which is the default branch, TeamCity doesn't detect a change on the other branch. This seems broken given how git manages branches.

0
Comment actions Permalink

Hi codekaizen.

Your issue seems slightly different to the issue described by Martin. I have replicated step by step what Martin mentioned and it has worked just fine in 10.0.4 (triggered build in dev via VCS trigger, during the build process checkout + rebase onto master, push, master gets triggered as well). Could you describe step by step how to replicate your issue?

0
Comment actions Permalink

Just to confirm. Does this happen as Martin mentioned only when you rebase while the previous build with the commit pre-rebase is being built? Or does it also happen every time you do a rebase? I have been running several tests today with rebasing, and they seemed to trigger a new build every time just fine, including fast-forward rebases.

On the other hand, runBuildOnSameRevisionInEveryBranch defaults to true, so I'd suggest to take it out.

In your situation, which is the default? master or release? I'm trying to replicate your scenario to see where the problem resides.

0
Comment actions Permalink

I'm having the same issue described in this thread at my company. We commit to a develop branch, then when we are satisfied with testing, do a PR/merge to a release branch. TeamCity is set to monitor both branches. If we do a squash merge, it does trigger a build in release but we've recently switched to just merge and TeamCity does not pick up on the change and build the release branch, nor does it show any changes pending. If I manually kick off a build in release by clicking the run button, the resulting build does have all the latest changes, but the Team City UI never reflects this.

I've played around with the teamcity.vcsTrigger.runBuildOnSameRevisionInEveryBranch setting. I've omitted it completely, set it to true and set it to false and it doesn't seem to change the behavior.

We are currently running version 2017.2.2 (build 50909) and are using GitHub Enterprise.

0
Comment actions Permalink

Having the same problem. We create tags in git, then run a build on that release tag, but the `build.vcs.number` is sometimes NOT the commit SHA for that tag, but one earlier. TeamCity even refuses to show the last commit that actually reflects the release tag.

https://youtrack.jetbrains.com/issue/TW-10084 seems to be related - but it's only open 13 years...

0
Comment actions Permalink

Hello,

> We create tags in git, then run a build on that release tag

How do you run a build on a tag? Did you enable "Use tags as branches" in Git VCS root and then just run a build on a tag as if it was a branch, or do you use some other approach?

Also what TeamCity version do you use?

0
Comment actions Permalink

Yes, exactly. We have this in the VCS settings:

 

And then select the tag:

 

---

We are currently on 2021.2.3.

0
Comment actions Permalink

So when the build is executed, what revisions are shown in the "Revisions" section of the Changes tab? Does the revision there correspond to the selected tag?

0
Comment actions Permalink

No:

 

This just happened to be commit from a settings change in TeamCity itself. But we have the same problem with other commits.

0
Comment actions Permalink

Could you please attach a build log of this build? Or at least the first part of it where TeamCity computes revisions.

0
Comment actions Permalink

Yes sure:

Build 'XXX / Build / Sources Container Core' #386, branch 'release_01.256.001'
Triggered 2022-06-21 15:11:20 by 'Snapshot dependency; Niklas Correnz (niklas.correnz); XXX / Deploy / Deploy Local Production, build #24'
Started 2022-06-21 15:11:35 on agent 'xxxlinux45-a'
Finished 2022-06-21 15:19:19 with status NORMAL 'Success'
VCS revisions: 'xxx_Git' (Git, instance id 988): '8e23b89ef44542c3d0149bf0b0b2133134cd3ba7' (branch: 'refs/tags/release_01.256.001', checkout rules: '+:. => project; -:.teamcity; -:tests')
TeamCity URL https://teamcity.local.xxx/viewLog.html?buildId=236806&buildTypeId=xxx_Build_SourcesContainerCore 
TeamCity server version is 2021.2.3 (build 99711), server timezone: GMT (UTC)

[15:11:20]W: bt397 (7m:59s)
[15:11:20]i: TeamCity server version is 2021.2.3 (build 99711)
[15:11:20] : Collecting changes in 1 VCS root
[15:11:20] :     [Collecting changes in 1 VCS root] VCS Root details
[15:11:20] :         [VCS Root details] "Git" {instance id=988, parent internal id=65, parent id=xxx_Git, description: "https://gitlab.local.xxx/xxx/webapp.git#refs/heads/master"}
[15:11:21]i:     [Collecting changes in 1 VCS root] Detecting changes in VCS root 'Git' (used in 'Build sources only', 'Create Tag' and 9 other configurations)
[15:11:21]i:     [Collecting changes in 1 VCS root] Will collect changes for 'Git' starting from revision 1a7ea926fd785ed6bfc8427f89aca06f67f16dab
[15:11:21] :     [Collecting changes in 1 VCS root] Compute revision for 'Git'
[15:11:21] :         [Compute revision for 'Git'] Upper limit revision: 1a7ea926fd785ed6bfc8427f89aca06f67f16dab
[15:11:21]i:         [Compute revision for 'Git'] MaxModId = 129459
[15:11:21] :         [Compute revision for 'Git'] The first revision that was detected in the branch refs/tags/release_01.256.001: 1a7ea926fd785ed6bfc8427f89aca06f67f16dab
[15:11:21] :         [Compute revision for 'Git'] Latest commit attached to build configuration (with id <= 129459): 8e23b89ef44542c3d0149bf0b0b2133134cd3ba7
[15:11:21] :         [Compute revision for 'Git'] Verified that 8e23b89ef44542c3d0149bf0b0b2133134cd3ba7 is the latest commit affecting the build by the checkout rules: +:. => project
-:.teamcity
-:tests
[15:11:21] :         [Compute revision for 'Git'] Computed revision: 8e23b89ef44542c3d0149bf0b0b2133134cd3ba7

 

It's the same with merge commits. Huge merge commits are ignored because TeamCity doesn't detect any changes and then the vcs number is wrong.

0
Comment actions Permalink

This one is from when it failed with a merge commit - almost identical log:

 

Build 'XXX XXX / Build / Sources Container Core' #360, branch 'release_01.253.000'
Triggered 2022-06-07 15:22:23 by 'Snapshot dependency; Niklas Correnz (niklas.correnz); XXX XXX / Deploy / Deploy Local XXX Production, build #21'
Started 2022-06-07 15:22:36 on agent 'XXXlinux45-a'
Finished 2022-06-07 15:30:09 with status NORMAL 'Success'
VCS revisions: 'XXXXXX_Git' (Git, instance id 988): 'fb3f805d0da5780196c12ab33f2f4ad5eb16c243' (branch: 'refs/tags/release_01.253.000', checkout rules: '+:. => XXX; -:.teamcity; -:tests')
TeamCity URL https://teamcity.local.XXX.de/viewLog.html?buildId=235618&buildTypeId=XXXXXX_Build_SourcesContainerCore 
TeamCity server version is 2021.2.3 (build 99711), server timezone: GMT (UTC)

[15:22:23]W: bt397 (7m:46s)
[15:22:23]i: TeamCity server version is 2021.2.3 (build 99711)
[15:22:23] : Collecting changes in 1 VCS root
[15:22:23] :     [Collecting changes in 1 VCS root] VCS Root details
[15:22:23] :         [VCS Root details] "Git" {instance id=988, parent internal id=65, parent id=XXXXXX_Git, description: "https://gitlab.local.XXX.de/XXX-XXX/webapp.git#refs/heads/master"}
[15:22:23]i:     [Collecting changes in 1 VCS root] Detecting changes in VCS root 'Git' (used in 'Build sources only', 'Create Tag' and 9 other configurations)
[15:22:23]i:     [Collecting changes in 1 VCS root] Will collect changes for 'Git' starting from revision 6173f6223381b0a0e70e3e38cc9ab7bdb1384285
[15:22:23] :     [Collecting changes in 1 VCS root] Compute revision for 'Git'
[15:22:23] :         [Compute revision for 'Git'] Upper limit revision: 6173f6223381b0a0e70e3e38cc9ab7bdb1384285
[15:22:23]i:         [Compute revision for 'Git'] MaxModId = 128989
[15:22:23] :         [Compute revision for 'Git'] The first revision that was detected in the branch refs/tags/release_01.253.000: 6173f6223381b0a0e70e3e38cc9ab7bdb1384285
[15:22:23]i:         [Compute revision for 'Git'] Check graph edges starting from fb3f805d0da5780196c12ab33f2f4ad5eb16c243 reachable from 6173f6223381b0a0e70e3e38cc9ab7bdb1384285
[15:22:23]i:         [Compute revision for 'Git'] Checked edges: 128987:1
[15:22:23] :         [Compute revision for 'Git'] Latest commit attached to build configuration (with id <= 128989): fb3f805d0da5780196c12ab33f2f4ad5eb16c243
[15:22:23] :         [Compute revision for 'Git'] Verified that fb3f805d0da5780196c12ab33f2f4ad5eb16c243 is the latest commit affecting the build by the checkout rules: +:. => XXX
-:.teamcity
-:tests
[15:22:23] :         [Compute revision for 'Git'] Computed revision: fb3f805d0da5780196c12ab33f2f4ad5eb16c243

0
Comment actions Permalink

Seems like checkout rules affect the calculation of the revision. There are exclude rules:

-:.teamcity
-:tests

In this case the computed revision will be the last one which affects the checkout directory, in other words the last one which affects any folder other than .teamcity or tests.

You can make TeamCity to publish additional parameters with prefix teamcity.upperLimitRevision. which will correspond to the latest commit affecting the VCS repository branch. For this you need to add the following configuration parameter to the build configuration where you need this behavior:

teamcit.internal.buildRevisions.publishUpperLimitRevisions=true

0
Comment actions Permalink

You are correct for the first posted case, the changed files are excluded via checkout rule, but I still want the correct commit in that case.

But in the case of a merge commit (2nd posted log), there are tons of files that are definitely not excluded, but TeamCity still doesn't see them:

 

________________________

 

I've tried your suggestion with the upper limit parameter, but all of these combinations are not working, the values remain empty:

 

(If I don't give them an empty default value, the configuration won't run at all. Says then there are no compatible build agents.)

0
Comment actions Permalink

Sorry there was a typo in the configuration parameter name, it should be:

teamcity.internal.buildRevisions.publishUpperLimitRevisions

If you run a build with this configuration parameter you'll see on the Parameters tab that there are new parameters with names: teamcity.upperLimitRevision.<VCS root id>

But apparently, these parameters are so "internal" that it's not possible to reference them. We'll fix it in the upcoming bugfix release: 2022.04.2.

Speaking about merge commits, in fact if merge commit affects files matched by the checkout rules, then a new build should run on the merge commit revision. If you see that TeamCity took some other revision instead, then then you can try to run the following git command:

git diff --names-only <merge commit revision> <revision of the build>

It will show the names of the files affected by the merge commit. If there are file names which are matched by the checkout rules, then this is a bug.

However, you're running a relatively recent TeamCity version and bugs like this should not happen anymore. There was quite a siginificant effort to eliminate them. In the build log you can actually see lines like:

 Verified that fb3f805d0da5780196c12ab33f2f4ad5eb16c243 is the latest commit affecting the build by the checkout rules: +:. => XXX
-:.teamcity
-:tests

This is TeamCity running something similar to git diff --names-only to check with the Git that the computed revision is correct.

0
Comment actions Permalink

Well, I don't doubt that TeamCity is doing everything correct with its diffs.

The simple problem remains that a git tag identifies a specific commit SHA. Not some abstraction that might be relevant to TeamCity.

If I cannot get that SHA from TeamCity, then I can't use the `vcs.number` for anything that leaves TeamCity. Like my docker images that should get tagged with the SHA of the git tag. Other software in our infrastructure expects that SHA to be the exact git tag.

I'm eagerly awaiting the release 2022.04.2 now :)

0
Comment actions Permalink

Hello,

2022.04.2 has been published yesterday. 

0
Comment actions Permalink

Yes, thanks, I've noticed today! We've upgraded and teamcity.upperLimitRevision works well! Thank you.

0

Please sign in to leave a comment.