Checkout directory marked for deletion
Hello,
Basically, our issue is that we have checkout directories that can take up to 300Gb of space and our agents only have a 400Gb disk. So whenever a checkout directory is marked for deletion (when requesting a clean build for instance), the entire directory will be copied to TC\work\.old before being deleted whenever the agent becomes idle. We obviously don't have enough space to have 2 directories of 300Gb so we end up with errors because the disk is full.
I can workaround this issue by providing a custom checkout directory which seems to cause a different behavior where the directory gets deleted directly instead of being copied to .old. However, I would prefer to avoid using custom checkout directories if possible.
Is there a way to tell TeamCity to not copy the checkout directory to .old before deleting it?
Thanks!
Michael
Please sign in to leave a comment.
Hello Michael,
Could you please:
1) confirm the TeamCity version in use, and
2) upload a build log for a build which failed because of the issue + agent logs for the build`s agent to https://uploads.jetbrains.com/, and share the upload ID with me? I see that, at least on the latest versions, TeamCity is moving the data, not copying it, but it could be that while the .old still remains (it is cleared in background), the checkout is already started and the agent machine runs out of disk space.
Hi Fedor,
We're using version TeamCity Enterprise 2019.2.1 (build 71758)
Here's the upload ID: 2020_11_25_GBfSuu9hnN3RWawm
This is exactly what is happening yes, the workspace still exists in .old while a new one is being checkout so we run out of disk space. So the question is: Can we configure TeamCity so it doesn't copy/move the directory to .old and simply delete it before the build starts?
I've read about the "Free disk space" feature but I'm not sure if that would help in this case. The doc mentions that it will try to clean "the checkout directories that were marked for deletion" before proceeding. I assume this includes the folders in .old? If so, that might be a solution!
Thanks!
Michael
Hello Michael,
Thank you for the logs! Yes, Free Disk Space build feature would work in this case; it uses a number of cleaners one of which is handling the .old directories. Could you please try to set this up to see if this solves your issue?
I have reached out to the development team to discuss if the logic may be amended so that TeamCity would try to get rid of .old before checkout; as soon as I have an update, I will get back to you.
Hello Michael,
Just a quick follow-up; do you have either teamcity.agent.build.checkoutDir.expireHours agent property, or system.teamcity.build.checkoutDir.expireHours build parameter active for the build/agent in question? (see the https://www.jetbrains.com/help/teamcity/build-checkout-directory.html#Automatic+Checkout+Directory+Cleaning for details)
Hi Fedor,
No I don't have either of these properties set up currently. We will be implementing the "Free Disk Space" feature to solve our issue.
Thanks!
Michael
old thread, but I found the real answer to this problem. I wish you guys added it to your official documentation instead of just listed in a comment in some random issue(https://youtrack.jetbrains.com/issue/TW-44974/Build-moves-checkout-directory-to-.old-instead-of-deleting-it-at-the-beginning-of-a-build-with-clean-checkout-free-disk-space#focus=Comments-27-1657299.0-0):
Set the following in the buildAgent.properties file:
teamcity.agent.use.delete.via.move=false