BuildAgent disk space check

One of the side effects of our increasingly popular TeamCity implementation is that our build agents are quickly running out of resources -- most notably disk space. To date we have about 115 distinct projects configured in TC (libraries, utilities, server-side apps, client-side apps, etc...) which at minimum have 2 build configurations each (a continuous build and a 'release candidate' build). Well done TC! :)
Pulling down a complete build workspace for each of these build runs, which for some projects also includes third-party .dll's and .so's to link against, starts to consume quite a bit of space.
I don't see anything in the agent properties that lists any real-time resource information about the box the agent is running in.
Granted, I can just throw more hardware at this problem but I rather be warned that a build agent is running out of space rather than act post-mortem after a build has failed due to disk-space.
I think a build agent property that specifies the current available disk-space would be very useful. That way a build agent can suddenly become "incompatible" with a build run and the build queue can take over and route the build to another, better suited, build agent.
This is of high priority to our org. so I'm also willing to do this as a plugin. Unfortunately, I don't see anything in the plugin API that might allow me to tack this on. Suggestions?

Thanks,

- Rene

9 comments

Rene,

Thank you for describing your case with agent running out of disk space.

Frankly, we sometimes face alike problems in out internal installation of TeamCity and too think how to cope with this. We tend to think that one of future TeamCity versions should support some kind of agent state reports with necessary notifications.

As to reporting disk space via agent properties. This is possible via plugin (actually our .Net runners report properties related to .Net Framework installed on the agent). The limitation here is that the properties are passed from agent to the server on not regular basis and currently we cannot guarantee server has the most current agent properties.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Hello Yegor,

we've been facing such problems as well, and think that a good solution might be to make the agent
automatically clean up unused work-folders (in our case, this is the cause for running out of disk
space) until enough space is available. Would that be possible?

Sascha

0

Hello Yegor,

we've been facing such problems as well, and think
that a good solution might be to make the agent
automatically clean up unused work-folders (in our
case, this is the cause for running out of disk
space) until enough space is available. Would that be
possible?

Sascha


Hi,

I would be against this. Sometimes, after a build has failed, we may want to logon to the agent to examine the work directories. Having the agent delete "unused" folders would hamper this.

Thanks,

Scott

0

Sascha, Scott,

Actually, we've done some in this direction:
On new build run, build agent checks for last time the directories were used for a build, and if the last build occurred more then some limit (2 weeks by default) ago, the directories are deleted. The deletion process tries to do this asynchronously, if possible.

This scenario seems to be suitable in most cases.
What do you think?

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Glad to finally see some discussion on this topic! :)

I agree with Scott in Quebec, we do not want to routinely clean up the work folder every time prior to a build run. If this was the feature that was added then I would insist in some control in how to configure this policy. i.e. I would want to specify the rules by which something is or is not a candidate for cleanup.

Yegor, as for the plugin route... are you saying I would have to use a custom build runner everywhere I wanted to do this check? That seems complicated because out of all my build configurations I have a variety of build runners: VS 2003, 2005, msbuild, nant, etc... so it would be a big project to replace all of this for every build just so that I can get this information. Or maybe I'm not understanding your example. Could you elaborate on that?

- Rene

0

Scott, Yegor,

of course I meant not to unconditionally clean up everything, but only in case not enough space and no other build agent is available: "... automatically cleanup unused work-folders ... until enough space is available" - i.e. only clean up if a build is certainly failing due to insuffient space (which can be caused by the output from previous builds).

Cleaning up until "enough space is available" is difficult though - nobody knows how much data a build produces - at least initially. A sophisticated approach could be to keep some statistics about the amount of data a build usually generates and use that as a hint. Even if it's not used for not cleaning up, that information could also be used to indicate whether a build-agent is capable of running a build or not - which at least would reduce those ugly "Build failed. No space left on device" error mails.

I'm not sure if the approach to only remove (very) old build results will work for us - a bunch of nightly builds can easily produce so much data that this could cause a build to fail as well - but it would be start anyway.

Sascha

0

But this logic will become very difficult to implement right away. Too many scenarios. I still think a build agent property and threshold is the best way to go. Just add a requirement to all your build configurations to require system.diskSpace >= 4GB and you're done. This way, at least you'll know you need to clean space in your Build agents when your build queue goes nowhere because it can't find any compatible agents.

0

Hello Rene,

The easyest way to solve you proble is to update ]]>/conf/buildAgent.properties

file from the other process/build/plugin. This change cause agent restart
and reconnect to
TC server thus all changes properties will be published.

It is possible to write plugin for the agent. It should be the class implementing
jetbrains.buildServer.agent.AgentPlugin interface (or extending
jetbrains.buildServer.agent.AbstractAgentPlugin)

But using plugins will allow you to run your code as a part of Agent JVM.
It is possible to define properties
only on the agent start.

If you have any questions please feel free asking it!

--
Eugene Petrenko
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

One of the side effects of our increasingly popular TeamCity
implementation is that our build agents are quickly running out of
resources -- most notably disk space. To date we have about 115
distinct projects configured in TC (libraries, utilities, server-side
apps, client-side apps, etc...) which at minimum have 2 build
configurations each (a continuous build and a 'release candidate'
build). Well done TC! :)

Pulling down a complete build workspace for each of these build runs,
which for some projects also includes third-party .dll's and .so's to
link against, starts to consume quite a bit of space.

I don't see anything in the agent properties that lists any real-time
resource information about the box the agent is running in.

Granted, I can just throw more hardware at this problem but I rather
be warned that a build agent is running out of space rather than act
post-mortem after a build has failed due to disk-space.

I think a build agent property that specifies the current available
disk-space would be very useful. That way a build agent can suddenly
become "incompatible" with a build run and the build queue can take
over and route the build to another, better suited, build agent.

This is of high priority to our org. so I'm also willing to do this as
a plugin. Unfortunately, I don't see anything in the plugin API that
might allow me to tack this on. Suggestions?

Thanks,

- Rene



0

Rene,

No, custom build runner is not needed. This can be simple agent-side plugin that will add agent properties. The current limitation is that plugin-provided properties are not meant to change over time, as Eugene noted.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Please sign in to leave a comment.