Are cloud based build agents guaranteed to only be executing one build at any given time?

Hi,

Are cloud-based (EC2) agents guaranteed to only be executing one build at any given time?

I am asking this, because for some reason, my Swabra cleaners often can't delete a whole bunch of files:

[16:22:36][Swabra] Scanning directory /home/svc_buildagent/buildAgent/work for newly created, modified and deleted files comparing with snapshot 6e52ff80.snapshot, paths to monitor are /home/svc_buildagent/buildAgent/work, /home/svc_buildagent/buildAgent/work/8bea4d1260be35ed
[16:22:36][Swabra] Detected modified /home/svc_buildagent/buildAgent/work/directory.map
[16:22:36][Swabra] Detected new, unable to delete /home/svc_buildagent/buildAgent/work/.old/1acf664ae1b5fdd4_3/packages/FAKE/FAKE.nupkg
...
[16:32:25][Swabra] Detected new and deleted /home/svc_buildagent/buildAgent/work/8bea4d1260be35ed/docker-gc-state
[16:32:25][Swabra] Detected 7011 unchanged, 7347 newly created (355 of them deleted), 477 modified, 1 deleted files and directories

When this happens, the clean up step usually takes up to 9 minutes.

My goal is to make Swabra delete all the files under /home/svc_buildagent/buildAgent/work (is this a good idea?) so that I always have enough disk space left before I start the next stage of my build pipeline.

Also when a build creates some artifacts, where do those artifacts get saved in? On the build agent itself? In which path? Can I configure TeamCity to store the artifacts in an S3 bucket?

Thanks,
Behrang

9 comments
Comment actions Permalink

Artifacts are published on server and saved under the server data directory.
Build agents do execute one build at a time, but if your build starts some processes and does not properly stop them, then such processes can block cleanup of files in checkout directory. You can configure Swabra to kill such processes automatically.
Also it is not a good idea to use Swabra for cleaning agent work directory. This directory contains some files which can be changed by build agent only, and removing of these files can cause unpredictable results.
Swabra should be used to clean files inside checkout directory only, this is what it is designed for.

0
Comment actions Permalink

Hi Pavel,

EDIT 1:

Also it is not a good idea to use Swabra for cleaning agent work directory. This directory contains some files which can be changed by build agent only, and removing of these files can cause unpredictable results.
Swabra should be used to clean files inside checkout directory only, this is what it is designed for.

Is there a way to configure Swabra to clean all checkout directories, not the one that is configured for the currently executing build?

Original post:

Also for some reason after configuring Swabra, there were still checkout directories that were not getting cleaned. To be more precsie, I had a project that had several build configurations in a pipeline connected through snapshot and/or artifact dependencies.

For example:

01 - Build
02 - Test
03 - Deploy

I added a step named "00 - Clean Agent" that had swabra as well as a few other scripts (i.e. https://github.com/spotify/docker-gc) running. However after a while I was seeing checkout directories that had lots of C# or Node JS libraries underneath them that were left uncleaned. And for some reason rerunning "00 - Clean Agent" was not cleaning them up.

Any ideas what I might have done wrong?

Thanks,
Behrang

Message was edited by: Behrang Saeedzadeh

0
Comment actions Permalink

Sorry I am new to TeamCity. Does "Artifacts are published on server and saved under the server data directory." mean that artifacts are not saved on the agents but the "master" TeamCity server?

Thanks,
Behrang

0
Comment actions Permalink

Hi, why do you want to clean all checkout directories?

Swabra is not designed for cleaning other checkout directories, but rather for cleaning out files created by the build. Its primary purpose is to ensure that checkout directory is clean and doesn't contain any traces of the previous build.

0
Comment actions Permalink

Artifacts are kept on agent as well for a short period of time, like 2 weeks, but they are also always uploaded to server. And server is the primary place to get artifacts from.

0
Comment actions Permalink

Yes, this is correct, artifacts are stored on the master server.

0
Comment actions Permalink

Hi Sergey,

Hi, why do you want to clean all checkout directories?

 


When a build is finished on an agent, is there any benefit in keeping its checkout directory on the agent -- especially if we have configured the build to clean the checkout directory before each run?

Screen Shot 2015-08-19 at 10.25.22 pm.png

The problem that we have is that over time our agent runs out of disk space. I am eventually going to increase its disk space size, but my thinking was that after a build is finished, its checkout directory is no longer needed. So I wanted to configure Swabra to clean all checkout directories as the first step in all my build pipelines.

Cheers,
Behrang

0
Comment actions Permalink

So in general our agents should have enough disk space for two weeks, correct?

Also in our case, we use cloud agents and I think they all get terminated after an hour of no use. So I guess we only need to make sure we have enough disk space for 1 day as usually they don't get used at night.

0
Comment actions Permalink

TC Agent is smart and tries to clean up its directory, if there's not enough free space. By default, it's required to have at least 3GB of disk space before running the build, otherwise TC will clean artifacts cache, other checkout directories, VCS cache, etc.

You can read more about checkout directory and cleaning concepts here: https://confluence.jetbrains.com/display/TCD9/Build+Checkout+Directory

Basically, you don't need to worry that BuildAgent will consume all free space. If there's enough space for running one build of one configuration, you don't have to worry. But please note that it's not applying to external caches, like global temp directories, local maven repositories, etc,

0

Please sign in to leave a comment.