Is there a way to compare two agent configurations?

Created January 19, 2022 01:08

I've got a main build server and an agent. Both appear to be configured identically, as I set them up at the same time. They have the same software installed with the same settings, and all of the Windows SDKs match. They were both building the build configuration in question at one time.

On the main build server, our project still compiles normally. On the other agent, we now just get errors like this:

fatal error C1083: Cannot open include file: 'new.h': No such file or directory

I've tried to use ProcessMonitor to look for clues about when it wants to compile new.h, to no avail. I've also confirmed that the two computers have the same environment variables.

As I stated before, what's interesting is that the agent *used to* work about 3 months ago. So it's like there's something the TeamCity configuration that changed that affects only that agent. However, I don't know how to perform some sort of TeamCity configuration comparison between the two, so I'm hoping someone here has suggestions for me.

Thanks!

11 comments

Guilherme Barbosa

Created February 02, 2022 09:11

Hello Dave,

If you only have two agents, then it's likely the issue is that those agents are on different machines that have different software (sdks, libraries, etc) installed. It is also possible that the agents might also be running on different paths and have something hardcoded, or that the first agent might not be using clean checkout so at some point it might have downloaded/created something and it started working for the most part, then it suddenly had a clean one and started failing due to that.

So checking these possibilities first might be a good idea, as probably there aren't so many configurations yet on those agents.

In any case, the following plugin Build Agent Diff might be useful for you to find differences between your Build Agents:

https://plugins.jetbrains.com/plugin/8997-teamcity-build-agents-diff

As a side note, there is no main build server: when you install TeamCity via the windows installer, the installation includes a buildAgent. However this is not a recommended setup for production purposes because of security concerns:

https://www.jetbrains.com/help/teamcity/security-notes.html#Recommended+Security+Practices

Please let us know if these suggestions help you or if you have any further questions

Thank you,

Guilherme

Dave Matsumoto

Created February 02, 2022 14:23

Hi Guiherme, thank you for your reply! I did check all of the installed system packages to ensure everything is the same, and to the best of my abilities, I determined that the two PCs are identical. I will do a clean build next on the working server, just to see what happens. I don't know why I didn't try that earlier!

I'll also give the tool a try, and will read up on the security issues you've mentioned, although our server is internal only and isn't exposed to the internet directly.

Thank you very much!
Dave

Dave Matsumoto

Created February 07, 2022 16:09

Hi again, Guiherme. I installed the plugin last week and did the diff. This is a very nice plugin! However, it did not reveal any differences that would affect the build, as far as I could tell:

working server has jre1.8.0_291, while the non-working one has jre1.8.0_311
working server has two extra folders in its PATH that are unrelated to the error I am seeing during the build.

I decided to brute force the build process by running Process Monitor and looking for the file referenced by the error message. I realize this in no way falls under the purview of "TeamCity support", but I was hoping you might have some feedback as to why I see the difference that is ultimately related to my problem.

On the working server, I can see that during compilation, the file "new.h" is accessed, as shown by the following lines:

9:14:32.5924063 AM    CL.exe    21248    QueryDirectory    C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\ucrt\new.h    SUCCESS    FileInformationClass: FileDirectoryInformation, Filter: new.h, 2: new.h
9:14:32.5925143 AM    CL.exe    21248    CreateFile    C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\ucrt\new.h    SUCCESS    Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, AllocationSize: n/a, OpenResult: Opened
9:14:32.5925858 AM    CL.exe    21248    ReadFile    C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\ucrt\new.h    SUCCESS    Offset: 0, Length: 1,416, Priority: Normal
9:14:32.5930662 AM    CL.exe    21248    QueryDirectory    C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\include\vcruntime_new.h    SUCCESS    FileInformationClass: FileDirectoryInformation, Filter: vcruntime_new.h, 2: vcruntime_new.h
9:14:32.5931551 AM    CL.exe    21248    CreateFile    C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\include\vcruntime_new.h    SUCCESS    Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, AllocationSize: n/a, OpenResult: Opened
9:14:32.5932093 AM    CL.exe    21248    ReadFile    C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\include\vcruntime_new.h    SUCCESS    Offset: 0, Length: 5,263, Priority: Normal
9:14:32.5936934 AM    CL.exe    21248    CloseFile    C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\include\vcruntime_new.h    SUCCESS    
9:14:32.5943682 AM    CL.exe    21248    CloseFile    C:\Program Files (x86)\Windows Kits\10\Include\10.0.18362.0\ucrt\new.h    SUCCESS

On the non-working server, I just see this error, with no attempted access of that same file:

c:\program files (x86)\microsoft visual studio\2017\professional\vc\tools\msvc\14.16.27023\atlmfc\include\afx.h(62): fatal error C1083: Cannot open include file: 'new.h': No such file or directory

By any chance, would you have a recommendation for me as to what I can try next?

Thank you,

Dave

Guilherme Barbosa

Created February 08, 2022 09:31

Hello,

It seems that in the non-working machine some components might be missing. I would suggest you to rerun the Visual Studio Installer, select Individual Components and reinstall all the necessary components:

MSVC;
C++ ATL;
C++ MFC;

Let me know if this works for you.

Thank you

Guilherme

Dave Matsumoto

Created February 08, 2022 16:28

Thanks, Guilherme, it appears that way, but the machine is set up correctly. Its Visual Studio setup is also identical to the working server.

In addition, the non-working server was fine 4-5 months ago, which is why I was wondering about an agent diff tool, thinking that one of us inadvertently messed up a setting. Also, Visual Studio can build the solution just fine on the non-working server -- it just won't build in TeamCity.

Guilherme Barbosa

Created February 10, 2022 10:18

Hi David,

Please check if the following guide helps you finding the cause of your problem:

https://www.jetbrains.com/help/teamcity/common-problems.html#Build+works+locally+but+fails+or+misbehaves+in+TeamCity

Thank you,

Guilherme

Dave Matsumoto

Created February 10, 2022 16:01

Thanks for the link, Guilherme. I'd already done that test -- my msbuild command works fine from the command prompt. TeamCity runs as the same user I login as locally.

Just so it's clear what I used -- here's my command that worked. I executed it from the VS2017 command prompt:

msbuild "path\to\my\solution.sln" /p:Platform=Win32 /p:Configuration=Release /m:8 /target:Rebuild

And this is how my project is configured in TeamCity:

Both working and non-working agents run the TeamCity service as the same domain user.

Guilherme Barbosa

Created February 15, 2022 23:10

Hello,

To verify if the build works the same in Team City as it does locally, can you try to run the command with which you run your build locally with the command line runner instead of the .NET runner? While . NET runner would do nearly the same thing as you did locally, it might not do exactly the same and it can help you finding out if the local build is using the same sources and working directory as TeamCity does.

Thank you,

Guilherme

Dave Matsumoto

Created February 16, 2022 06:25

Thanks for that suggestion, Guilherme. I should have tried that before. Well, it actually worked! So, I guess my next question is, what should I look at next, to understand why the .NET runner isn't working?

Guilherme Barbosa

Created February 16, 2022 08:48

Hello,

That means that the .NET runner is using different sources and/or working directories. As such, you can check under agent details page -> parameters and compare with what is being used in your local build, and also the .NET runner build step configuration to ensure it uses the same sources/working directories.

Thank you,

Guilherme

Dave Matsumoto

Created February 16, 2022 14:01

I suppose comparing logs is a good way to determine some of those details. I firmly believe that the .NET runner is using the same sources, as the checkout directory is what's used in both configurations. The agent system properties and environment variables are the same between build servers, but I'll inspect the configuration parameters more closely, as there are several in the list. Thank you!

Please sign in to leave a comment.