[SOLVED] Build Tags Created with Service Messages

Created August 17, 2023 20:22

Using the Service Message "addBuildTag" I am adding a tag during a build. This works just fine as long as the build is running, but once the build is complete it is being removed from the build. In the log I do see the message "Tag MyTag added to the build via a service message". And it is there, until the build completes, at which point it disappears. This is actually the same behavior as manually tagging. Why don't the tags survive the completion of the build? Is there a way to prevent this?

In contrast, if I tag a build after it is complete, the tag does not go away.

11 comments

Anatoly Cherenkov

Created August 21, 2023 15:42

Hi! There could be some script that removes the tag (by default, TeamCity doesn't do that). You can check the teamcity-activities.log to see which user removed the tag. Look for lines like this:

[2023-08-21 17:40:54,307] INFO - s.buildServer.ACTIVITIES.AUDIT - build_promotion_tag_change: Build BUILD_PROMOTION{id=66346}'s tags were changed by "user with id=1" with comment "Removed: test-tag"

Kaboomer13

Created August 21, 2023 22:02

Thanks for the help Anatoly!

Unfortunately, I can't see anything in the log file removing the tags. This is what I do see:

[2023-08-21 15:56:26,345]   INFO - tbrains.buildServer.ACTIVITIES - Build started; Utility Builds / Agent Builds / Test {id=UtilityBuilds_AgentBuilds_Test/bt839} #5 {id=316960, agent="elastic_mclaren/172.27.190.38" {id=46859}, triggered by "user with id=4" (##userId='4' type='user'). Started 2023-08-21 15:56:26.335, running}
[2023-08-21 15:56:39,518]   INFO - tbrains.buildServer.ACTIVITIES - Tag TagTest2 added to the build #5 {build id=316960, buildTypeId=UtilityBuilds_AgentBuilds_Test} via a service message
[2023-08-21 15:56:39,523]   INFO - tbrains.buildServer.ACTIVITIES - Tag TagTest1 added to the build #5 {build id=316960, buildTypeId=UtilityBuilds_AgentBuilds_Test} via a service message
[2023-08-21 15:56:52,452]   INFO - s.buildServer.ACTIVITIES.AUDIT - build_promotion_tag_change: Build BUILD_PROMOTION{id=316960}'s tags were changed by "user with id=4" with comment "Removed: TagTest1, TagTest2"
[2023-08-21 15:56:52,456]   INFO - s.buildServer.ACTIVITIES.AUDIT - build_promotion_tag_change: Build BUILD_PROMOTION{id=316960}'s tags were changed by "user with id=4" with comment "Added: CustomTagTest, TagTest1, TagTest2"

Here you can see the addition of "TagTest1" and "TagTest2" during the build, then a Remove/Add was me adding another tag named "CustomTag2". There are no other items in this log (which is the tail log for teamcity-activities) mentioning tags.

I've reviewed our plugins and I don't see any that should be doing anything fancy with tags, and this build is a new configuration just for testin this, so I didn't do anything special there.

Any other insight would be greatly appreciated, thanks.

Anatoly Cherenkov

Created August 28, 2023 16:20

I am testing it on 2023.05.3 by adding a tag via REST API (https://www.jetbrains.com/help/teamcity/rest/manage-finished-builds.html#Manage+Build+Tags) in a build step, but the tag persists after the build is complete. Could you explain in a bit more detail how exactly you add the tag and in which TeamCity version?

Kaboomer13

Created August 30, 2023 23:37

I am running TeamCity Professional 2023.05.1 (build 129321). I am creating the tags using the service message (from PowerShell here):

Write-Host "##teamcity[addBuildTag 'TagTest2']"

One factor that may impact this is that the builds are running in a Docker cloud agent running Windows. I have also attempted to use the REST api to create tags with the same result, unfortunately.

Anatoly Cherenkov

Created September 07, 2023 11:10

Hi! I didn't test it with Docker cloud agent, but it works fine for me with a K8s cloud agent. Could you run a dummy build with this step:

Write-Host "##teamcity[addBuildTag 'TagTest-%teamcity.build.id%']"

Then confirm the tag disappeared after the build finished and upload these logs:

full build log (go to the Build Log page > Download log > select Full build log or Zip).
All <TeamCity home directory>/logs/teamcity-server.log* files.
All <TeamCity home directory>/logs/teamcity-activites.log* files.

Feel free to upload the files through https://uploads.jetbrains.com/ (don't forget to specify the upload ID).

Kaboomer13

Created September 07, 2023 15:26

I've followed your steps with a brand new build configuration. This configuration only has a single PowerShell step with the single command you provided copied in. This is what the build page looks like after the build is complete:

And this is what it looks like if I refresh the page. The tag is gone:

However, if I add one manually and refresh, it is persistent:

I've uploaded your requested log files with this ID: 2023_09_07_FiqHQCEFUJmL7piPcN45yr

Anatoly Cherenkov

Created September 14, 2023 18:47

Unfortunately, the logs didn't reveal the cause. Does the manually added tag stay if you add it while the build is running? You might need to add a `sleep X` command to the testing configuration, or otherwise, the build may finish too quickly.

Kaboomer13

Created September 14, 2023 18:53

I have indeed tried that, and no, manually created tags that are created before the build completes do not survive completing either.

Anatoly Cherenkov

Created September 15, 2023 13:07

If there are any external (non-bundled, and especially 3rd-party) plugins, I'd recommend disabling those, restarting the server, and checking if the issue reproduces. I don't see any other potential reasons why a tag would be removed.

Kaboomer13

Created September 15, 2023 15:18

Wow! That solved it. I had reviewed the external plugins, but concluded that there was nothing that *should* be interacting with tags. I disabled all of these (not a very good diagnostic technique, I know):

Auto Properties 0.6 by Jon Philpot
VCS Support: CVS SNAPSHOT-202202022202 by JetBrains
Microsoft Teams Notifications 2.0.3 by Spyder007
Kubernetes support snapshot-20200127161010 by JetBrains
TestingBot 1.0 by TestingBot
Url build trigger 231 by JetBrains, s.r.o.

My best guess is the CVS support or perhaps TestingBot, since they may have features to tag builds. I managed to get in this morning before builds started, but now there are builds running so I'd rather not test further. If you need I can do so over the next couple days though.

Thank you very much for your help on this. I have several builds that really benefit from tags.

Anatoly Cherenkov

Created September 18, 2023 08:30

Thanks for confirming it. I narrowed it down to the TestingBot plugin and filed this issue in the plugin repository: https://github.com/testingbot/testingbot-teamcity-plugin/issues/1.

Please sign in to leave a comment.