distributing long-running builds

We are deploying a long-running build configuration to TeamCity. How will TeamCity distribute it to agents if we allow it to run on any agent?

In a fresh workspace,

  • a clean checkout takes 15 min
  • a clean build takes 90 min

In contrast, incrementally we have

  • a checkout at 1 min
  • an incremental build with the median of 10 min


Obviously, there's a strong incentive to rerun the build in a previously used workspace (105 min vs. 11 min).

Now suppose such a build is triggered, and TC has to decide:

  • to run the build immediately, in a clean workspace (105 min)
  • to wait for a "scandalous" 30 min and run it in a previously used workspace (30 + 11 = 41 min)

Will it correctly pick the latter?

I guess for this to be possible, TC would have to collect the history of (A) clean and (B) incremental checkout/build times separately. Does it?

Thanks,
Sam

9 comments
Comment actions Permalink

I'm not sure I understand the setup. Are you going to clean the checkout directory before each build or not? If you are not then the long build times you've presented are only relevant in preliminary builds, unless you have too many build agents and some don't run the build very often. The simple solution here is to limit the agents you allow to build the configuration to just as many as it takes to not get behind over the course of the week. If you start seeing the build queue add up, just throw another agent at it.

To answer your question, I think it only tracks build step and test times along with the total run time. Although it's not a long shot to get the sync times if you have times for everything else.

0
Comment actions Permalink

Hi Mitchell,

Thanks for your response.

We're not actively cleaning the check-out directories. TC will clean them up for us, though; we have build configs with huge disk consumption and there's no way each agent can accommodate all the workspaces. As a result, TC automatic clean-up is bound to kick in at some agents. At that point, we would like to see our long-running build config to (strongly!) gravitate towards an agent where a clean check-out isn't necessary.

@JetBrains, can you comment on the query above?

Thanks,
Sam

0
Comment actions Permalink

JetBrains,

Any comments, please?

Thanks,
Sam

0
Comment actions Permalink

JetBrains,

This should be a pretty simple answer; can you respond please?

Thanks,
Sam

0
Comment actions Permalink

I don't see them replying so I thought I'd offer a suggestion, if I may. Is it possible to divide up the build tasks so you don't have tons of trees on the same machines. Take for example:

Let's say I have 12 VCS roots to build from and 4 build agents. If I divide the builds into one configuration per VCS root and delegate which build agents can run which builds (by using pools) I can reduce the number of builds each agent is allowed to do to only 3. Not only will I reduce the amount of build time require per change but I'll increase agent reuse and it should sync much less often and much faster.

0
Comment actions Permalink

Thanks, Mitchell. Unfortunately, we can neither divide up the build tasks, nor can we have one build configuration per VCS root.

0
Comment actions Permalink

Hello, Samuel!

Sorry for delayed response! As to your query: nope, current implementation does not allow to have builds postponed in the queue while there is any compatible agent being ready to execute it right now.
In the case TeamCity choosing between those two agents both being ready to run build right now, it obviously picks up the fastest. However we're tackling that kind of problem right now.

0
Comment actions Permalink

Alexey,

Thanks. Can you please be more specific? Are you going to implement something similar to what I describe? Would that be in scope for 8.0?

Thank you,
Sam

0
Comment actions Permalink

Not exactly: we're working on advanced approaches to distributing builds across the agent-set, however, those would hardly account for the details of the cases like clean/incremental patch/build execution. And yet there is no particular milestone.

The easiest, and unfortunately sole, workaround for your case, is to partition your agent-set by introducing specific agent-requirement like agent-name, etc. targeting particular agents, so that having 'bulky' run-configurations to be separated at 'agent-set' level. That should help you preserve your agents from superflous contention.

0

Please sign in to leave a comment.