Upgrade to TeamCity 8.1.1 produces some weird behavior for our builds

Hello everyone,

After upgrading our server to 8.1.1 and having the build agents upgrade and come back we noticed some strange behavior in some of our builds. We have a couple of 4.5 builds and some legacy 3.5 builds, they all worked flawlessly on TeamCity 8.0.x.The .NET 4.5 builds are not having any issue whatsoever, they all built fine, deploys are done, excellent, but for the legacy projects they just fail to build.

After inspecting the build logs for the failing builds they all had a similar error message:

   SGEN An attempt was made to load an assembly with an incorrect format: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscorlib.dll.

And for some projects they were failing on:

    SGEN An attempt was made to load an assembly with an incorrect format: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Data.dll.

I then disabled all our agents but one and enabled debug logging on it, after checking the XMLRPC log I found a suspicious entry:

    <param name="MSBuildTools4.0_x64_Path" value="c:\Windows\Microsoft.NET\Framework64\v4.0.30319" />
    <param name="MSBuildTools4.0_x86_Path" value="c:\Windows\Microsoft.NET\Framework64\v4.0.30319" />

and also this one

   INFO - cher.DotNetPropertiesExtension - Found MSBuild Toolset 4.0 (x86) at c:\Windows\Microsoft.NET\Framework64\v4.0.30319

Would this imply it's going for 64-bit files even when x86 is set? Anyway, we did a full clean on the agent and tried again, now it was failed on a different csproj file with a new error, stating:

    ASPNETCOMPILER error ASPRUNTIME: The current identity (<domain>\BuildService) does not have write access to 'C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files'.

Why would it try to write there? We have no 64-bit 2.0 projects, only 32-bit, I verified this from inspecting the XML files that were generated by doing a backup before the upgrade, they clearly state:

    MSBuild = NET Framework 4.0, ToolsVersion = 3.5, Platform = x86;

After the upgrade they are now:

    MSBuild = NET Framework 4.0, ToolsVersion = 4.0, Platform = x86;

Then looking at the earlier snippet from the log, I would almost think that something is forcing my build to use the 64-bit files of the Framework, when everything in the TeamCity configuration is set to "x86" and also in our csproj files the platforms are explicitly set to "x86". I am baffled for the moment what exactly is going on. I will edit this post as I dig further and find more information that could help you assist us with our friday morning headache. It's entirely possible that I am interpreting the logfiles wrong but regardless we're still stuck with these few builds.

[BeginEdit]

Found some additional difference between the build log of yesterday (8.0.5) and today (8.1.1):

TeamCity 8.0.x - Command Line

    C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\sgen.exe /assembly:C:\Projects\Camden\Components\ParkMobile.Business\obj\Release\Parkmobile.Business.dll /proxytypes /reference:C:\Projects\Camden\References\Aspose.BarCode.dll /reference:C:\Projects\Camden\References\externalreferences\Aspose\v6.6\Aspose.Words.dll /reference:C:\Projects\Camden\References\CarlosAg.ExcelXmlWriter.dll /reference:C:\Projects\Camden\References\NHibernate\Castle.Core.dll /reference:C:\Projects\Camden\References\CipherLite.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\fluentnhibernate-NH3.1-1.2\FluentNHibernate.dll /reference:C:\Projects\Camden\References\Log4Net\log4net.dll /reference:C:\Projects\Camden\References\MichaelBrumm.Globalization.SimpleTimeZone.dll /reference:C:\Projects\Camden\References\MichaelBrumm.Win32.TimeZones.dll /reference:C:\Projects\Camden\References\EntLib_4.1\Bin\Microsoft.Practices.EnterpriseLibrary.Caching.dll /reference:C:\Projects\Camden\References\EntLib_4.1\Bin\Microsoft.Practices.EnterpriseLibrary.Common.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\fluentnhibernate-NH3.1-1.2\NHibernate.ByteCode.Castle.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\NHCH-3.1.0.GA-bin\SysCache\NHibernate.Caches.SysCache.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\fluentnhibernate-NH3.1-1.2\NHibernate.dll /reference:C:\Projects\Camden\Components\ParkMobile.Core\bin\Release\Parkmobile.Core.dll /reference:C:\Projects\Camden\Components\ParkMobile.Data\bin\Release\Parkmobile.Data.dll /reference:C:\Projects\Camden\References\Parkmobile.RdwService.dll /reference:C:\Projects\Camden\Components\ParkMobile.Resources\bin\Release\Parkmobile.Resources.dll /reference:C:\Projects\Camden\References\externalreferences\StructureMap2.6.1\deploy\StructureMap.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Configuration.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Data.DataSetExtensions.dll" /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Data.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Drawing.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.EnterpriseServices.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Messaging.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.0\System.Runtime.Serialization.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.0\System.ServiceModel.dll" /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Transactions.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Web.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Web.Extensions.dll" /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Web.Services.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Windows.Forms.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Xml.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Xml.Linq.dll"

TeamCity 8.1.x - Command Line

    C:\Program Files\Microsoft SDKs\Windows\v7.1\bin\sgen.exe /assembly:C:\Projects\Camden\Components\ParkMobile.Business\obj\Release\Parkmobile.Business.dll /proxytypes /reference:C:\Projects\Camden\References\Aspose.BarCode.dll /reference:C:\Projects\Camden\References\externalreferences\Aspose\v6.6\Aspose.Words.dll /reference:C:\Projects\Camden\References\CarlosAg.ExcelXmlWriter.dll /reference:C:\Projects\Camden\References\NHibernate\Castle.Core.dll /reference:C:\Projects\Camden\References\CipherLite.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\fluentnhibernate-NH3.1-1.2\FluentNHibernate.dll /reference:C:\Projects\Camden\References\Log4Net\log4net.dll /reference:C:\Projects\Camden\References\MichaelBrumm.Globalization.SimpleTimeZone.dll /reference:C:\Projects\Camden\References\MichaelBrumm.Win32.TimeZones.dll /reference:C:\Projects\Camden\References\EntLib_4.1\Bin\Microsoft.Practices.EnterpriseLibrary.Caching.dll /reference:C:\Projects\Camden\References\EntLib_4.1\Bin\Microsoft.Practices.EnterpriseLibrary.Common.dll /reference:C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\fluentnhibernate-NH3.1-1.2\NHibernate.ByteCode.Castle.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\NHCH-3.1.0.GA-bin\SysCache\NHibernate.Caches.SysCache.dll /reference:C:\Projects\Camden\References\externalreferences\NHibernate\FluentNHibernate\fluentnhibernate-NH3.1-1.2\NHibernate.dll /reference:C:\Projects\Camden\Components\ParkMobile.Core\bin\Release\Parkmobile.Core.dll /reference:C:\Projects\Camden\Components\ParkMobile.Data\bin\Release\Parkmobile.Data.dll /reference:C:\Projects\Camden\References\Parkmobile.RdwService.dll /reference:C:\Projects\Camden\Components\ParkMobile.Resources\bin\Release\Parkmobile.Resources.dll /reference:C:\Projects\Camden\References\externalreferences\StructureMap2.6.1\deploy\StructureMap.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Configuration.dll /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll" /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Data.DataSetExtensions.dll" /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Data.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Drawing.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.EnterpriseServices.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Messaging.dll /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0\System.Runtime.Serialization.dll" /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0\System.ServiceModel.dll" /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Transactions.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Web.dll /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Web.Extensions.dll" /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Web.Services.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Windows.Forms.dll /reference:C:\Windows\Microsoft.NET\Framework64\v2.0.50727\System.Xml.dll /reference:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Xml.Linq.dll"

Notice how the 8.1.x log is showing references to Framework64 and to C:\Program Files\ as opposed to the 8.0.x log showing references to Framework and to C:\Program Files (x86)\;

Hope this helps!

[EndEdit]

Thanks in advance, TeamCity rocks!

2 comments
Comment actions Permalink

Would be great if someone from Jetbrains or the TeamCity team could reply to this post as right now my perspective is that there is a possible bug in TeamCity 8.1.1. in regard to using MSBuild 4.0. If this is due to another cause it would be great if anyone can confirm that as well so I don't have to keep my hopes up and try to find an answer from a different approach.

In the meantime I am going to change the MSBuild of each build step to 4.5, but still with a Target Framework of 3.5 to see if this fixes something, since all the proper 4.5 builds have no issues. Then again the MSBuild 4.5 paths also don't have the Framework64 notice in their path from the build agent configuration debug dump.

[BeginEdit]

Just checked the agent logs/configuration on our other build server, which is still running 8.1 and the same path deviation is immediately apparent:

TeamCity 8.1

MSBuildTools4.0_x64_Path c:\Windows\Microsoft.NET\Framework64\v4.0.30319
MSBuildTools4.0_x86_Path c:\Windows\Microsoft.NET\Framework\v4.0.30319


TeamCity 8.1.1

MSBuildTools4.0_x64_Path c:\Windows\Microsoft.NET\Framework64\v4.0.30319
MSBuildTools4.0_x86_Path

c:\Windows\Microsoft.NET\Framework64\v4.0.30319


Hope this helps.

[EndEdit]

Thanks again!

0
Comment actions Permalink

Registered an issue in the bug tracker: TW-35530 - MSBuild Toolpath possibly corrupted after upgrading to 8.1.1

0

Please sign in to leave a comment.