TeamCity MSBuild step is hanging since updating to TeamCity Professional 2025.03

We recently updated to TeamCity Professional 2025.03 (Build 186049) as well as updated our Java to v21 on the build agents. Since then on our macOS build agent, we've been experiencing issues building our Xamarin.iOS project.

First, our restore nuget step seemed to time out suddenly, which at least threw an error. This has previously not been an issue, but the fix was to replace the 

nuget restore MySolutionName.sln

to 

msbuild MySolutionName.sln /t:restore /p:RestorePackagesConfig=true /p:Configuration=Release /p:Platform=iPhone

But now the bigger deal is that the build step for generating the IPA is hanging and I've pretty much exhausted my current options for solving this. 

So the build step is this:

msbuild /p:Configuration=Release /p:Platform=iPhone /p:BuildIpa=true /target:Rebuild MySolutionName.sln

And I have let this run for quite a while, it does seem like it gets stuck and hangs at the same points. Then if you let it run for a while, it progresses slightly, but I have yet to see it finish. And usually after an hour hanging on the step, I stop the process. 

I want to note that I have run these exact commands locally on the build server in the terminal and they have completed successfully. And the IPA is generated.

This is the behavior I observe:

So each time it will get stuck first on CoreCompile. It was also hanging with “Using shared compilation with compiler from directory: /Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/Current/bin/Roslyn” logged, so I tried adding /p:UseSharedCompilation=false to the build step, but no luck. It still gets stuck instead on CoreCompile. 

After a while, you will get stuck on just attempting to create directories. 

Creating directory "bin/Release/".
Creating directory "obj/Release/".

The furthest I have gotten it is to:

Tool arch execution started with arguments: -arch arm64e /usr/bin/xcrun ibtool --errors --warnings --notices --output-format xml1 --minimum-deployment-target 17.2 --target-device iphone --auto-activate-custom-fonts --sdk /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS17.2.sdk --compilation-directory

Now I am still targeting Xcode 15.2 to try and squeeze out a few more Xamarin builds before it is no longer accepted, and I am working on a dotnet version of our applications. But it would be nice to be able to get this working and like I said, I am 100% it works locally on the build server, just not when run through TeamCity currently.

I ran  top -o cpu and observed that mono-sgen64 was consuming high CPU during the hang.

And I will attach the agent thread dumps to this post as well from during the hang.

Any help on this issue would be greatly appreciated!

Upload id: 2025_03_26_Rbp45ntyrCkbMtkkAs5Pdi (file: my_tc_logs.zip)