Triggering builds: Catch-22

Hi

I have what seems to be a catch-22 situation with a set of build configurations I'm managing. Our TeamCity environment is completely virtualised, with each build agent running in a separate VM. This set of build configurations deletes the build agent VMs on a weekly basis, and then recreates them from a number of different templates. The way I have it configured is as follows:

  1. Run "Delete Existing Build Agents". This removes all build agent VMs, except for the agent currently running the job.
  2. Run each of the following in parallel:
    1. "Regenerate Build Agents (template 1)"
    2. "Regenerate Build Agents (template 2)"
    3. "Regenerate Build Agents (template 3)"
    4. "Regenerate Build Agents (template 4)"


My initial configuration had the four "Regenerate Build Agents" configurations set up with a Finish Build trigger on "Delete Existing Build Agents", but unfortunately this lead to problems where one of the build configurations would try to create an agent with the same name as the one left over from step 1. To get around this, I reconfigured the "Regenerate Build Agent" configurations to use snapshot dependencies on "Delete Existing Build Agents". This worked fine for the first week or two.

Unfortunately this week the job failed because the "Delete Existing Build Agents" configuration hadn't run - the dependencies were configured to only trigger it if there was no suitable build available. With no changes made in the last week to the scripts governing this process, last week's run was deemed suitable, and no agents were deleted. As a consequence, the four "Regenerate Build Agents" configurations each failed because they were trying to create agents that already existed.

I don't believe that I can configure this dependency to run the "Delete Existing Build Agents" configuration regardless of whether or not there is a suitable build, as I believe this will result in four runs of this configuration being triggered. Potentially, that could be disastrous: we might end up with no agents left at all.

What I am looking for is a way to reconfigure these builds to run so that just one "Delete Existing Build Agents" is run for all four of the "Regenerate Build Agents", that this configuration is always run, and for the whole thing to run on a weekly schedule. Is it possible to configure these build configurations in this manner? Perhaps I need to restructure the configurations to support this chain better - if so, how?

Thanks in advance

Alastair

3 comments
Comment actions Permalink

Hi Alastair,

Not sure I undestand the case with deleting and regenerating agents in full, so it might make sense to reformulate this in generic terms, detailing the behavior you need to get.

Using snapshot dependencies with "Do not run new build if there is a suitable one" turned OFF seems the most appropriate setup for snapshot dependencies.
You might need to select the agent to run the maintenance tasks and do not schedule any taks on it. Deletion and recreation of the agnet can then be done after another agent is reinitialized, with dedicated set of the build configuraitons.

Actually, you might consider adding one more agent whch would be reserved for this kind of tasks. Or performing the tasks from another machine, where TeamCity build would just trigger the script start, if at all.

0
Comment actions Permalink

Hi Yegor

Thanks for your reply.

The general situation that I have is one of a two-step pipeline/chain, where the second stage has a number of configurations running in parallel. Something like this:

    +---B1
   /

  / +---B2
/ /

A-+
\ \
  \ +---B3
   \

    +---B4

All five configurations must run on each overall run, and each of the B configurations must run after the A configuration has run. The A configuration must run only once, and B3 and B4 must run on the same agent as A. If it helps to think of it as steps in a software build process, you could think of A as a Clean build, and each of the Bs as compilation builds of separate components.

What I can't figure out is the correct dependency and trigger configuration between A and each of the Bs. My concern about turning off "Do not run new build if there is a suitable one" is that a separate A build will be triggered for each of the Bs.

My original intention was to have a separate agent dedicated to the purpose of tearing down and creating new agents, but this didn't go down well with people who didn't like the idea of having an agent licence being unused for all but 3-4 hours/week.

0
Comment actions Permalink

Hi Alastair,

> All five configurations must run on each overall run, and each of the B configurations must run after the A configuration has run. The A configuration must run only once, and

FOr this, you can add a "master/top" build configuration wich would have snapshot dependencies on all of Bs and trigger this master build configurations, not A or Bs.

> B3 and B4 must run on the same agent as A.

This can be achieved by "run on the same agent" in the snapshot dependency settings.


As to triggering, here is a relevant note form the doc:

> When setting up triggers for the builds in the chain, the recommended approach is: think about the result - the build you want to get at the end of the process, and configure  triggers in its corresponding, "top" build configuration. No triggers  are necessary in the build configurations this top one depends on, as  their builds will be put into the queue automatically when the top one  is triggered.

0

Please sign in to leave a comment.