Build configuration has increased time
Our configuration cleans the directory and pulls the repo, then uses MSBuild steps to build different pieces of software. This usually would take about 36 minutes in all, 4 of that for the repo pull. For the last 5 weeks it is averaging around 5 hours, the repo portion taking 13 minutes. Sometimes we've seen it even higher, the peak at 7 hours and 47 minutes.
No changes have been made to the build configuration or even any significant changes to the repo. There are other build configurations that haven't really increased their times too much. They don't pull the repo, and have only varied by a small amount (~4 minutes to ~6 minutes, ~24 minutes to ~31 minutes).
I ran the MSBuild command from the command line (only tested one project so far) and it took the normal average amount of time, so we're led to believe this is exclusively a TeamCity related issue, but don't know how TC could be the cause of the problem. Any suggestions?
Please sign in to leave a comment.
Can you please share build logs for any older build that was running normally, and for any newer build that is delayed so I could compare these? You may upload them via uploads.jetbrains.com site - just let me know the upload ID once this is done.
Unfortunately, I don't think I will be able to do that. It contains company confidential information. Is there something else that I can do to help?
You could fetch two files on your own and use any text file comparing tool so to try and pinpoint what sections/lines are exactly causing troubles, and then share just these removing/replacing the confidential details.
So the two biggest points are the checkout and the MSBuild. The specific line on the checkout is
"C:\Program Files\Git\bin\git.exe" -c core.askpass=C:\BuildAgent\temp\buildTmp\pass4861669642793686695.bat -c credential.helper= -c credential.helper=C:/BuildAgent/temp/buildTmp/credHelper2003811745916785241.bat checkout -q -f master
And the MSBuild is a little more complicated, the nested steps are MSBuild > Solution.sln: Build target: Build > Build > MSBuild > Project.vcxproj: Build default targets > ResolveProjectReferences > MSBuild > CommonLibrary.vcxproj: Build default targets > ClCompile > CL. Specifically that is the second of four CL steps. Nested inside of this is a list of C++ files and each of those will take around 30 seconds or so. Before, they only took a second at the most. There are 29 different solutions that are being built in this configuration, so I only provided a sample of one, but they all use the same configuration, just a different solution.
Were there any changes to the agent machines, namely, the update of OS version or perhaps installation/upgrade of AV software in use? Recently, we had a similar case where VS tests were severely affected by an antivirus tool, and were substantially faster when it was disabled on the machine - what makes it semi-similar to your issue is the fact that loading `.pdb` files was delayed there, and in your case it sounds like the same issue affects the sources themselves. I am just trying to make sure that there were no recent environment changes that could have affected the build.
The only recent changes that we made were installing Docker Desktop and a Linux distro. But the performance of the machine was really bad when trying to do anything with the distro, so we uninstalled it. Left Docker because it's needed for other tasks, but it's not actually running any containers. I also tried shutting down Docker and running the build and it made no difference.
I also want to reiterate that when I manually ran the build command via command line the build time was back down to its normal length. Specifically when TeamCity issues the command is it taking a long time.
Speaking of MSBuild command, how is it started (via a command line runner or via a specific runner - if yes, which one exactly)? Also, can you please confirm if your manual ran was on the same agent machine under the same user, or on your local machine?
Via the MSBuild runner. The manual run was on the same agent and user.