What is this .old directory and why is it devouring my agent?

Some of our builds have started failing due to insufficient space on the agent machines. Investigating it appeared the checkout directories were being copied to a .old directory inside work. Why is this happening and how can I stop it? There's nothing helpful in the documentation.

System details:

Teamcity 5.0.3 on ubuntu 8.04, Agents on Red Hat EL 5, SuSE11, Windows 2003 and Solaris 10

0
9 comments

Jennifer,

This works as follows. If agent understands that checkout directory is no longer used, agent moves it under the .old folder. Then, when agent becomes idle, it will clean files from this folder. But if agent never becomes  idle,  it  is possible that .old will increase and only when available disk spac reaches some limit, agent will try to remove .old during the next build start.

If you have problems with disk space, then probably cleaning logic is not working properly. Please provide agent logs.

Kind regards,
Marina

0

Hi,

I have same problem on Linux. A .old folder is now 1To and no space left. I've clean it up myself, but I don't understand what could be wrong, and what exactly are these folders used for?
And can we configure the cleanup, or see cleanup logs somewhere?

0

Hello,

As long  as  you do not run out of disk space, you can leave the decision  to  clean  .old to an agent itself. But if you have problems with disk space, then probably cleaning logic is not working properly. In this case please send as agent logs.

Kind regards,
Marina

0

I'm having the same issue, where do I send the log files, and which ones do you want?

0

We'll need \buildAgent\logs\teamcity-agent.log
You can post it here, or send to teamcity-feedback@jetbrains.com (please mention a lint to this thread).

Thanks
Michael

0

The issue is eventually filed as TW-17652.

0

Although being 11 years old some question:
When does an agent becomes idle and will start cleanup? We are using cloud profiles and are suspending agents.
Or other way: how long should Cloud profile agents run so .old can be cleaned up?

2

In our case, if an agent runs several builds serially, there is no idle time between builds, then it causes the agent has no space since TeamCity doesn't have chance to remove .old directory.

Is there anyway to disable this .old function cause we don't need this backup folder and it caused space issue with bad impact.

2

I have this issue. Whenever I try to do a clean build it runs the build agent out of disk space. I don't have enough disk space for two whole copies of the checkout directory. Here is the relevant bit from the log from the build agent:

[2023-07-25 02:00:20,227]   INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.CreateCheckoutDirectoryStage
[2023-07-25 02:00:20,227]   INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.steps.RunAllBuildStepsStage
[2023-07-25 02:00:20,227]   INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.ClearRevisionInfoIfCleanBuildStage
[2023-07-25 02:00:20,242]   INFO -    jetbrains.buildServer.AGENT - Will clear revision info for checkout directory 'cef9b9e220fae651' and build id '1738'
[2023-07-25 02:00:20,242]   INFO -    jetbrains.buildServer.AGENT - Deleted 'cef9b9e220fae651.xml'
[2023-07-25 02:00:20,242]   INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.CleanBuildCheckoutDirectoryIfNeededStage
[2023-07-25 02:00:20,242]   INFO - CheckoutDirectoryIfNeededStage - Clean build enabled: removing old files from D:\BuildAgent\work\cef9b9e220fae651
[2023-07-25 02:00:20,258]   INFO - rectories.DirectoryCleanerImpl - Move directory D:\BuildAgent\work\cef9b9e220fae651 to D:\BuildAgent\work\.old\cef9b9e220fae651_1 for cleaning
[2023-07-25 02:00:20,258]   INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.CleanGeneratedFilesStage
[2023-07-25 02:00:20,258]   INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.buildStages.startStages.LogCheckoutDirectoryStage
[2023-07-25 02:00:20,258]   INFO -    jetbrains.buildServer.AGENT - Checkout directory: D:\BuildAgent\work\cef9b9e220fae651
[2023-07-25 02:00:20,258]   INFO - uildStages.BuildStagesExecutor - Call stage jetbrains.buildServer.agent.impl.patch.CheckoutSourcesBuildStage
[2023-07-25 02:00:20,258]   INFO - l.patch.AbstractSourcesUpdater - Updating sources: agent side checkout
[2023-07-25 02:00:20,273]   INFO -    jetbrains.buildServer.AGENT - Marking checkout directory cef9b9e220fae651 for build id 1738 as 'dirty'
[2023-07-25 02:00:20,273]   INFO -    jetbrains.buildServer.AGENT - No revision information for cef9b9e220fae651.xml on agent
[2023-07-25 02:00:20,273]   INFO -    jetbrains.buildServer.AGENT - Recording vcs state: revision: 87, VCS root signature: 559910aac4540b50, checkout rules: =>
[2023-07-25 02:00:20,273]   INFO -    jetbrains.buildServer.AGENT - Stored new configuration file cef9b9e220fae651.xml
[2023-07-25 02:00:20,273]   INFO -    jetbrains.buildServer.AGENT - Marking checkout directory cef9b9e220fae651 for build id 1738 as 'clean'
[2023-07-25 02:00:20,273]   INFO - pl.patch.ProjectSourcesOnAgent - Updating sources for root id=1; name=lawn_mowing_sim; revision: 87
[2023-07-25 02:00:20,351]   INFO - ernalArtifactsProcessingLogger - Publishing artifacts process finished
[2023-07-25 02:46:36,115]   WARN - l.patch.AbstractSourcesUpdater - Error while checkout on agent: Svn checkout: error for SVN path https://192.168.20.153:18080/svn/lawn_mowing_sim/trunk; Exit code: 1; svn: E155009: Failed to run the WC DB work queue associated with 'D:\BuildAgent\work\cef9b9e220fae651\Assets\Textures\Quarantine_Textures', work item 7289 (file-install Assets/Textures/T_HQ_Low_Asphalt1_Damaged1_1_BC.png 1 0 1 1)
svn: E720112: Can't write to stream: There is not enough space on the disk.  

0

Please sign in to leave a comment.