Agent is randomly cleaning checkout directory
One of our build agents has recently started randomly cleaning the checkout directory before a build. This is bad because it takes 5 hours and requires manual setup after sync completes! It has happened 3 times in the past 3 weeks.
I investigated the logs and tracked it down to this sectionINFO - jetbrains.buildServer.AGENT - No revision information for e02a450cf68b8bb7.xml on agentINFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.CleanBuildCheckoutDirectoryIfNeededStageINFO - CheckoutDirectoryIfNeededStage - Will perform clean checkout. Reason: [ROOT_DIRECTORY_WAS_CLEANED_UP_WITH_ERRORS]INFO - rectories.DirectoryCleanerImpl - Move directory C:\BuildAgent\work\e02a450cf68b8bb7 to C:\BuildAgent\work\.old\e02a450cf68b8bb7_0 for cleaningINFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.CleanGeneratedFilesStageINFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.LogCheckoutDirectoryStageINFO - jetbrains.buildServer.AGENT - Checkout directory: C:\BuildAgent\work\e02a450cf68b8bb7INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.patch.CheckoutSourcesBuildStageINFO - l.patch.AbstractSourcesUpdater - Updating sources: auto checkout (on agent)INFO - jetbrains.buildServer.AGENT - Marking checkout directory e02a450cf68b8bb7 for build id 19240 as 'dirty'WARN - jetbrains.buildServer.AGENT - old file e02a450cf68b8bb7.xml.old exists. Previous build might have not finished successfullyWARN - jetbrains.buildServer.AGENT - new file e02a450cf68b8bb7.xml.new exists. Previous build might have not finished successfully
This No revision information for part has occurred on all 3 clean builds.
Does anyone know what might be causing this?
In one case the previous job was aborted due to a system reboot, but in the other 2 cases the previous job completed successfully.
Please sign in to leave a comment.
Could you reproduce the build with unnecessary cleanup and attach the complete build log and teamcity-agent.log that covers the time the issue was reproduced?
You can also use the already existing build with this behavior if the teamcity-agent.log for this time period was not overwritten.
You can upload logs using https://uploads.jetbrains.com/ and share the upload ID.
Best regards,
Anton
Hi Anton,
Thanks for your response. UploadID: 2024_06_11_DhQS7hMbbsFL4DGWrt9F1W
I have disabled the agent, so the logs are for the last build that had the problem. It's very difficult to reproduce and has randomly happened only 3 times in the past 4 weeks.
If you search the teamcity-agent.log for “[2024-06-07 13:00:28,474]” then that's the build that caused the clean.
This behavior and these error messages mean that the agent does not know what revisions were checked out to the checkout folder, so the agent can't do an incremental update and has to perform a clean checkout.
It could happen in the following cases. Please check if one of the following applies to the other 2 cases not related to the reboot:
1. The version control settings could've been changed compared to a previous build of the same build configuration. In this case, a new hash code will be generated (which means a new checkout directory).
2. If the "clean build" setting is enabled for this build, then this information will be removed every time (most probably not the case, as it is random for you).
3. The agent may also lack disk space for other builds, which can lead to the removal of this information as well.
4. Also, if the agent fails to checkout files to this directory, it will remove this information because it no longer knows the state of the checkout directory.
The good news is that we have the option to disable clean checkout in such cases: https://www.jetbrains.com/help/teamcity/perforce-workspace-handling-in-teamcity.html#Forced+Protection+Against+Clean+Checkout.
Please see if it works for you.
Best regards,
Anton
Hi,
Looking at the log again, it seems that the agent was rebooted in the middle of running a job on the ‘main’ workspace. After rebooting the next job was on the tools only workspace which worked fine, then the next job on the ‘main’ workspace was the one that caused the clean due to
old file e02a450cf68b8bb7.xml.old exists. Previous build might have not finished successfullyI think this has happened in the past too (Windows Update!). So I don't think that our situation is one of the 4 options you mentioned, I think it just that the machine was rebooted while executing a job and this leaves the directory in an unknown state (leaving the xml file in place which causes the clean the next time).
I think in our case, ensuring Windows Update doesn't run, and I will set that clean checkout protection flag too just in-case.
Thanks, Noel
OK, please feel free to comment in the thread if it happens again for some other reason.
Best regards,
Anton