TeamCity do not detect changes in Git branches when merging / rebasing
Background:
We have the following set up with TeamCity Enterprise 8.1.1:
- One git repository with three branches:
- develop
- master
- release
- One TC project linked to a single VCS root:
- Default branch: refs/heads/develop
- Branch specifciation: +:refs/heads/*
In my TC project, I have a VCS trigger (default options, no filters)
If I make a change and push to develop, master or release, TC detects a change in develop and builds it just as expected.
Problem:
- I make a change to the develop branch
- I push that change
A build now starts for the develop branch. While this build is running: - I run git checkout master
- I run git rebase develop
- I run git push
At this point both develop and master points at the same change set in git.
At this point, I would expect a build to start on the master branch as well, but that does not happen. The build on the develop branch finishes a few minutes later, but nothing is built for master branch.
Some problem isolation:
- When I run the above steps, after step 5 TeamCity does not show any pending changes for the master branch. I expect changes to be shown since I just merged the develop branch into master and pushed.
- I've tried running git merge develop instead of git rebase develop in step 4 but that does not help.
- If I make my change locally both in master and in develop and then push both branches at the same time, then both branches are built.
- If TeamCity is busy and the build is queued in step 2 and I push while it's queued, then a build for both branches will be queued.
- If TeamCity starts to build the develop branch in step 2 and I stop this, then the master branch is built after step 5.
Please sign in to leave a comment.
I'm experiencing the same issue, is there any fix on this? I would expect a fast-forward rebase or merge to a branch to trigger a build regardless of whether that changeset have been built on a different branch.
I'm running 10.0.4 and have the same issue. When 2 branches point to the same changeset, one of which is the default branch, TeamCity doesn't detect a change on the other branch. This seems broken given how git manages branches.
Hi codekaizen.
Your issue seems slightly different to the issue described by Martin. I have replicated step by step what Martin mentioned and it has worked just fine in 10.0.4 (triggered build in dev via VCS trigger, during the build process checkout + rebase onto master, push, master gets triggered as well). Could you describe step by step how to replicate your issue?
The issue is that I have 2 git branches: master and a release branch. When the release branch and master branch point to the same changeset after a merge (i.e. the release branch was fast forwarded), TC does not detect changes on the release branch. I set the property teamcity.vcsTrigger.runBuildOnSameRevisionInEveryBranch to "true" but it appears to not have an effect. The only way to get TC to recognize changes on the release branch is to add another commit to just that branch.
Just to confirm. Does this happen as Martin mentioned only when you rebase while the previous build with the commit pre-rebase is being built? Or does it also happen every time you do a rebase? I have been running several tests today with rebasing, and they seemed to trigger a new build every time just fine, including fast-forward rebases.
On the other hand, runBuildOnSameRevisionInEveryBranch defaults to true, so I'd suggest to take it out.
In your situation, which is the default? master or release? I'm trying to replicate your scenario to see where the problem resides.
I have the same problem. If use release branches in git, and if I fast-forward merge to master (which has already been built) TeamCity doesn't trigger a build, and if I manually build it says there are no changes.
I'm having the same issue described in this thread at my company. We commit to a develop branch, then when we are satisfied with testing, do a PR/merge to a release branch. TeamCity is set to monitor both branches. If we do a squash merge, it does trigger a build in release but we've recently switched to just merge and TeamCity does not pick up on the change and build the release branch, nor does it show any changes pending. If I manually kick off a build in release by clicking the run button, the resulting build does have all the latest changes, but the Team City UI never reflects this.
I've played around with the teamcity.vcsTrigger.runBuildOnSameRevisionInEveryBranch setting. I've omitted it completely, set it to true and set it to false and it doesn't seem to change the behavior.
We are currently running version 2017.2.2 (build 50909) and are using GitHub Enterprise.
I'm facing similiar issues pushing different branches. Our workflow are basically the same described by Matthew. We have two branches, master and release. Master being the default branch. Development is made in master, and when we are satisfied with the changes, we do a merge/PR to branch release. We are currently using TeamCity 2017.2.2 (build 50909) and GitLab for VCS.
When we do some changes in release and then push, TeamCity builds correctly. Then we do a fast-foward to master and then push. TeamCity detects pending changes on master and then builds.
The problem occurs when branch master is pushed first. For example. We do some changes in master and push. TeamCity detects pending changes and then build. Next we fast-foward master to release and then push release. TeamCity does not find any pending changes in release.
Looking for help I found this information, which says:
For my understanding heres is where it lies my problem. Since master (default branch) was pushed first and release last changes came from fast-fowarding master, TeamCity will calculate the changes against master, the default branch, and not against release, the branch being pushed.
Similar issue here with v2019.1.2, although in our case the builds are triggering but the changes are not being tracked and reported correctly.
We have a develop branch, and periodically we'll do a fast-forward merge from develop to a QA branch (these merges only ever happen one-way).
Any PR that commits to develop first show up as "Pending" on the build config and then triggers a build in short order. While building, the changes are still visible on the dashboard and after building the changes reported for that build are correct.
Whenever we merge from develop -> QA, we first see the "Pending" changes for that branch on the build config, and the build is properly triggered. However, as soon as the build starts, the dashboard shows "No changes". After the build finishes, the dashboard still shows "No changes", as does the "Changes" tab for the build. What's stranger is that the "Changes" tab after the build does actually show a link to a "revision" (commit hash?) which when clicked shows the merge change details (files, comment and user).
How the hell is this still an issue, Im facing the same issue right now..
Having the same problem. We create tags in git, then run a build on that release tag, but the `build.vcs.number` is sometimes NOT the commit SHA for that tag, but one earlier. TeamCity even refuses to show the last commit that actually reflects the release tag.
https://youtrack.jetbrains.com/issue/TW-10084 seems to be related - but it's only open 13 years...
Hello,
> We create tags in git, then run a build on that release tag
How do you run a build on a tag? Did you enable "Use tags as branches" in Git VCS root and then just run a build on a tag as if it was a branch, or do you use some other approach?
Also what TeamCity version do you use?
Yes, exactly. We have this in the VCS settings:
And then select the tag:
---
We are currently on 2021.2.3.
So when the build is executed, what revisions are shown in the "Revisions" section of the Changes tab? Does the revision there correspond to the selected tag?
No:
This just happened to be commit from a settings change in TeamCity itself. But we have the same problem with other commits.
Could you please attach a build log of this build? Or at least the first part of it where TeamCity computes revisions.
Yes sure:
It's the same with merge commits. Huge merge commits are ignored because TeamCity doesn't detect any changes and then the vcs number is wrong.
This one is from when it failed with a merge commit - almost identical log:
Seems like checkout rules affect the calculation of the revision. There are exclude rules:
-:.teamcity
-:tests
In this case the computed revision will be the last one which affects the checkout directory, in other words the last one which affects any folder other than .teamcity or tests.
You can make TeamCity to publish additional parameters with prefix teamcity.upperLimitRevision. which will correspond to the latest commit affecting the VCS repository branch. For this you need to add the following configuration parameter to the build configuration where you need this behavior:
You are correct for the first posted case, the changed files are excluded via checkout rule, but I still want the correct commit in that case.
But in the case of a merge commit (2nd posted log), there are tons of files that are definitely not excluded, but TeamCity still doesn't see them:
________________________
I've tried your suggestion with the upper limit parameter, but all of these combinations are not working, the values remain empty:
(If I don't give them an empty default value, the configuration won't run at all. Says then there are no compatible build agents.)
Sorry there was a typo in the configuration parameter name, it should be:
teamcity.internal.buildRevisions.publishUpperLimitRevisions
If you run a build with this configuration parameter you'll see on the Parameters tab that there are new parameters with names: teamcity.upperLimitRevision.<VCS root id>
But apparently, these parameters are so "internal" that it's not possible to reference them. We'll fix it in the upcoming bugfix release: 2022.04.2.
Speaking about merge commits, in fact if merge commit affects files matched by the checkout rules, then a new build should run on the merge commit revision. If you see that TeamCity took some other revision instead, then then you can try to run the following git command:
git diff --names-only <merge commit revision> <revision of the build>
It will show the names of the files affected by the merge commit. If there are file names which are matched by the checkout rules, then this is a bug.
However, you're running a relatively recent TeamCity version and bugs like this should not happen anymore. There was quite a siginificant effort to eliminate them. In the build log you can actually see lines like:
Verified that fb3f805d0da5780196c12ab33f2f4ad5eb16c243 is the latest commit affecting the build by the checkout rules: +:. => XXX
-:.teamcity
-:tests
This is TeamCity running something similar to git diff --names-only to check with the Git that the computed revision is correct.
Well, I don't doubt that TeamCity is doing everything correct with its diffs.
The simple problem remains that a git tag identifies a specific commit SHA. Not some abstraction that might be relevant to TeamCity.
If I cannot get that SHA from TeamCity, then I can't use the `vcs.number` for anything that leaves TeamCity. Like my docker images that should get tagged with the SHA of the git tag. Other software in our infrastructure expects that SHA to be the exact git tag.
I'm eagerly awaiting the release 2022.04.2 now :)
Hello,
2022.04.2 has been published yesterday.
Yes, thanks, I've noticed today! We've upgraded and teamcity.upperLimitRevision works well! Thank you.