git tags as branches - changes incorrectly calculated?

Hello Oliver,

Yes, it appears that the issue is indeed linked to the logic behind the pending changes calculation. Whenever there is a new build, we are checking whether there are new changes or not by comparing to the previous build. As of now, we have two algorithms to locate previous build:
1) the last build on the default branch is being selected (default behavior);
2) the last build on any branch (date-wise) is being selected.

The last time when a default branch has been built would mean the last time there was a build ran on default branch.

When a build is located by the above algorithm, we are comparing current changes to the revision of that previous build. Any delta between the last state and current state is considered to be pending changes. As new tags become more and more different from the default branch, the list of changes would grow further because older relevant changes would be considered as not built (and pending) even though they were built. 

The possible workarounds are:

1) you may run build on default branch every so often; this could be feasible if you would tag sources after the build, with VCS Labeling feature (https://www.jetbrains.com/help/teamcity/vcs-labeling.html), so that you would always build a default branch and mark certain successfully built commits with tags;

2) alternatively, you may set this build parameter (https://www.jetbrains.com/help/teamcity/configuring-build-parameters.html):

teamcity.internal.vcsTrigger.runBuildOnSameRevisionInEveryBranch=false

What it would do is switch to the second algorithm (select last build on any branch instead of default). By that, you would ensure that most of the time last build would be more up to date and more likely that there are no pending changes.

I have also added an internal reference on your case to this issue: https://youtrack.jetbrains.com/issue/TW-66479, which is a request for optional third algorithm for pending changes calculation. If there is anything I could help with, please let me know.

Hi Fedor,

thanks for the detailed explanation.

Building the default branch and using the VCS Labeling feature isn't feasible in my workflow since not every commit is a release/should be tagged.

I tried using "teamcity.internal.vcsTrigger.runBuildOnSameRevisionInEveryBranch=false" in the "<Root project>", the actual Project, the Build and also in the agent it self but it didn't made any difference.

I also tried to build the default branch on every commit and skip the "release" build steps, only on the default branch, by using a condition, which kind of worked but had the downside that if i push the latest commit and set the tag a few minutes later, no new build would be triggered because there is no new commit (which makes sense but is also hindering me)-

Also sorry, i forgot to mention the installed version of TC: TeamCity Professional 2020.1.5 (build 78938)

Regards
Oliver

Hello Oliver,

Apologies, I forgot to mention that the VCS Labeling feature allows to label the sources manually as well (see https://www.jetbrains.com/help/teamcity/vcs-labeling.html#Manual+VCS+labeling for the details), so in theory you could move the tagging part into TeamCity but otherwise have manual control over when exactly the sources are tagged (and would also know for sure the sources were built successfully). Maybe this could work for you?

Speaking of the property; could you please confirm if on running the build the pending changes were calculated as if they were based on the default branch build instead of the latest date build?

Hi Fedor,

thanks for the info, i was able to find the option for manually tagging but i don't think that would be sufficient. I would like to patch the assembly version based on the tagged version number, sign the release binaries and upload them, which means i need to do a few more steps after the tagging as release.

Regarding the property: There where no difference in the calculation.

I did two commits and then tagged the latest commit, so every "version" contains two commits.
I also checked the "Parameters" tab in the builds to verify if the property was actually set.

Hello Oliver,

Sorry for the delay with the response here.
First of all, my apologies as I have not considered the scope of this property; it will affect only the builds which are started with VCS trigger. The builds following first one on a tag branch will behave properly, as there would be at least one build on the branch by then; manual builds would still include all changes not built on default branch yet. 

I have reached out to the development team to see if there is any logic to allow the same behavior for the default branches, but, unfortunately, there is none. The https://youtrack.jetbrains.com/issue/TW-66479 I have shared earlier would be the best shot to allow to do this preserving your current workflow; otherwise, there are two things I`d like to bring up:

1) you have mentioned there are other steps after tagging the build, but I do not quite follow the difference of manually tagging a build in TeamCity versus doing the same on VCS side. Could you please explain the key difference so I understand your use case better?

2) speaking of the default branch builds, there is an additional option; the following build parameter:

teamcity.vcsTrigger.runBuildInNewEmptyBranch = true

will, if set on the build configuration/project level, make the newly created tags trigger automatically even if there are no changes (compared to the default branch). 

Hi Fedor,

no worries, i'm glad that some one is helping me, even if i have to wait a day or two :-)

Not sure if i understand it correctly, but the builds in my screenshot are all done automatically and not manual, so shouldn't it work then with "runBuildOnSameRevisionInEveryBranch"? Or would it only consider the "changes" if there where a new automatically build in the respective tag/branch?

1) i try to automate my release steps which are: patching the assembly version in the code, build the code, sign the assemblies, create an msi installer, generate changelog, create a github release with the changelog, attach the msi file to the github release.
The version number for "patching the assembly version" is based on the tag (tag v1.0.1 would patch the assembly to 1.0.1).

The decision when to create a release is done while i'm in the cli and push the latest commit, usually i set a tag like v1.0.1 and then build the release etc. If i would use the VCS labelling option and tag the desired version as release (v1.0.1) manually, i would need to add a condition to every build step to exclude "<default>" except the "build the code" step and would have the issue that not every tag would get build (runBuildInNewEmptyBranch = false; need to check that parameter). Also this would be a manual process which i would like to avoid.

2) i tested the parameter and it seems to be working fine except that every tag/branch doesn't have any changes and i would need to add a condition to every build step, which i also would like to avoid.

Would it cause any issues if i would use my initial configuration (only build v* branches/tags, not master) and the changes counter would rise with every commit? Maybe performance or stability issues if the number gets to large?

Hello Oliver,

> the builds in my screenshot are all done automatically and not manual, so shouldn't it work then with "runBuildOnSameRevisionInEveryBranch"

Yes, it should; if the first build on tag branch is started via VCS trigger, then it will calculate own changes basing on the latest (date-wise) build in the configuration, regardless of the branch. The following builds will do the same if started via VCS trigger, and for the manual execution they will base off the existing build in the tag branch.

> The decision when to create a release is done while i'm in the cli and push the latest commit...

I see; a quick suggestion which comes to mind is to consider if this build can be split in two. The one part would be actual code build, which would produce the binaries; you could run it on any branch (default or tag) to produce the artifacts for the artifact dependency on the second part. The latter would handle only the release-specific steps and require artifacts produced by the former. The approach still leaves the manual tagging from TeamCity, though - but at least the condition requirement is skipped. 

> it seems to be working fine except that every tag/branch doesn't have any changes and i would need to add a condition to every build step

The newer builds will pick up the changes; it is just the chain of builds is now scattered over branches, so if you run VCS triggered builds exclusively, then you could track the chain of changes date-wise (across all the branches). 

> Would it cause any issues if i would use my initial configuration (only build v* branches/tags, not master) and the changes counter would rise with every commit?

While the relevant REST API calls would come more costly because of that, it would be significant only on the larger installation (e.g. with high frequency of commits and builds). This data is well-indexed, so if this installation is not too large or does not have many active contributors, I would not expect any troubles. Please mind that this is a theoretical consideration; as per my check, we did not test the relation between the number of build changes and the data processing.

I hope this helps; please let me know if there is anything else I could assist with.

Hi Fedor,

>> the builds in my screenshot are all done automatically and not manual, so shouldn't it work then with "runBuildOnSameRevisionInEveryBranch"

> Yes, it should; if the first build on tag branch is started via VCS trigger, then it will calculate own changes basing on the latest (date-wise) build in the configuration, regardless of the branch. The following builds will do the same if started via VCS trigger, and for the manual execution they will base off the existing build in the tag branch.

but it didn't worked for me as you can see in the screenshot, I created a completely new project, set the parameter, did the commits/tagged it and waited for the build to kick in.

> I see; a quick suggestion which comes to mind is to consider if this build can be split in two.

I tried that but had the same issue, since "<default>" never gets build in the "release" build configuration, the changes are piling up with every commit.

Guess currently, the only working solution, until you can properly configure the "changes calculation algorithm", is to set "teamcity.vcsTrigger.runBuildInNewEmptyBranch=true", add conditions to all build steps related to a "release" and include "<default>"  so that it always gets build.

Thanks for your time and help :)

Hello Oliver,

Again, sorry for the delay here.

> but it didn't worked for me as you can see in the screenshot
It is my understanding that tag branches were handling their changes correctly (I assume that builds #3, #5, #7 and #9 were based off the tag on the commit that was built in #2, #4, #6 and #8, respectively). Specifically, "even" builds were on master, which means that regardless of runBuildOnSameRevisionInEveryBranch value tag builds would base their changes off the latest build on master (if there would be only tag-based builds present and runBuildOnSameRevisionInEveryBranch was set to false, they would base their changes off the latest tag build). Could you please confirm the behavior you would have expected in this case?

>I tried that but had the same issue, since "<default>" never gets build in the "release" build configuration, the changes are piling up with every commit.
Yes, you are right; sorry, I have missed this detail.

Hi Fedor,

Sorry i guess we got mixed up a little bit.

The Screenshot with the master and tag builds (https://teamcity-support.jetbrains.com/hc/user_images/tMO__C_KKtyb1mUKtIky-g.png) are only with "teamcity.vcsTrigger.runBuildInNewEmptyBranch = true" and the screenshot with tags only (https://teamcity-support.jetbrains.com/hc/user_images/5EpqyXmlHFmW3QL6KLR1Mg.png) is with "teamcity.internal.vcsTrigger.runBuildOnSameRevisionInEveryBranch = false".

> if there would be only tag-based builds present and runBuildOnSameRevisionInEveryBranch was set to false, they would base their changes off the latest tag build

And that is what i tried initially, but it didn't worked for me, as you can see in https://teamcity-support.jetbrains.com/hc/user_images/5EpqyXmlHFmW3QL6KLR1Mg.png

For that screenshot i created a new project, added the "teamcity.internal.vcsTrigger.runBuildOnSameRevisionInEveryBranch = false" parameter, set the build branch filter to "+:v* <line break> -:<default>" (only build tags starting with "v", don't build "<default>") and then committed two changes to the git repository, waited for the trigger to kick in, all automatically (i did this multiple times to verify the result).

Hello Oliver,

Thank you for the clarification! That is definitely weird; I believe the logic around runBuildOnSameRevisionInEveryBranch was present for some while, but just in case, what version of TeamCity do you use? (I have been testing on 2020.1.5 and 2020.2). 
I will check with the development team in the meantime to see if there is relevant debug logging available.

Previously "TeamCity Professional 2020.1.5 (build 78938)" and now i'm running "TeamCity Professional 2020.2 (build 85487)"

Hello Oliver,

Just to double-check; while code-wise I believe it should be working even if added as a build or project parameter, do you see default behavior if the 

teamcity.internal.vcsTrigger.runBuildOnSameRevisionInEveryBranch=false

is added as an internal property? Alternatively, does it calculate changes properly in this setup?

Hi Fedor,

i configured the property, restarted the hole server, created a new project, configured and tested it: same result

Don't get confused by the commit message "test 11", i accidentally used the same commit message for both commits

Hello Oliver,

Thank you for the details (and for the prolonged patience too!). Development team has suggested to check certain part of VCS trigger debug which will appear in the teamcity-vcs.log if debug-vcs logging preset is on. If the above internal property is still on, could you please run the tag workflow one more time so that the changes are calculated incorrectly, and then share the log via https://uploads.jetbrains.com? (You may want to revert to default after that as the preset is particularly verbose.)

Hi Fedor,

sorry for the delay, got a little bit stressful at my end.

I've created a new project, configured it as previously and set the internal property "teamcity.internal.vcsTrigger.runBuildOnSameRevisionInEveryBranch=false" again.

Then activated the debug-vcs present and then made the first "version" (v1.0.0) at around 20:43:20 o'clock, waited for the automatic build to finish and created another version (v1.0.1) at around 20:45:57. Both commits/versions should only contain one change, but again, the second one shows the previously change as well.

Here is the corresponding log file: Upload id: 2020_12_16_QoAaY2qKMiRujU1B (file: teamcity-vcs.log)

Thanks again for your help :)

Hello Oliver,

Thank you! It is really appreciated - I have just checked the logic with our development team, and got a few updates:
1) we were able to reproduce the behavior (turns out my earlier local test was not exactly equal to the case you are having hence the difference in the output - sorry about that);
2) the logic is confirmed to work correctly for the feature branches, but the behavior is, unfortunately, different for tag references;
3) the team is discussing internally on how should the tags be handled going forward (e.g. should they behave the same as feature branches, reusing the already build commits, or not). 

As soon as I hear back on the latter, I will get back to you.

Hi Fedor,

thanks for the quick update, i really appreciate it.

Hello Oliver,

Just a few details on the issue. The property governs what changes are *included* into the build, but the representation of pending changes/changes that were included into a build uses a different algorithm and as of now there is no other configuration option to ensure changes from older tags would not appear on newer tags (unless there are corresponding default branch builds).
The https://youtrack.jetbrains.com/issue/TW-66479 I have shared earlier should help with the issue.

I am most sorry for the inconvenience this might be causing; if there is anything else I could help with, please let me know.

Hi Fedor,

again, thanks for your help, i will monitor the referenced ticked and in the meantime i guess i will build master and skip the "release" build steps (until there is a different way).

Have nice holidays and stay safe!

Hi Oliver,

I'm having pretty much the same problem. What workaround are you currently using?

My current way (to build a npm package) is to have a job that builds the branches just as usual. But when the master is built (e.g. after merging a branch), the job increments the version in package.json and creates an appropriate commit and tag. This works pretty nice, only drawback is that building version 2.0.25 has "Release version 2.0.24" as a change in its list.

Looking forward to hearing from you,

Sven

Hi Sven,
i added to all the buildsteps/runner a condition "teamcity.build.branch.is_default = false", this way the "master" branch gets checked out and thats it, the build for master is finished.

So all "changes" are calculated in the "master" branch and all my tags (v* branches) have "no changes" but are still getting build and then released to github.