Joining build dependency triggers and conditional builds

Hi

We are using TeamCity on our project and think its a really great product.  I have a couple of questions that it would be great to get some opinions on.

Currently we have a number of projects set up within team city.  We have many dependencies between these projects.  We have been using the build dependency triggers to manage these dependencies and overall it has been working well.

Now we have more of a dependency graph or tree of projects as opposed to a dependency chain or sequence.  Several of our components depend on a number of other projects.  What we would like to implement is a build trigger which would wait for all of its dependencies to have been updated before launching a build.  We were wondering if this is possible out of the box or if we need to upgrade to 5.1 and create our own custom build trigger.  Specifically we are finding that some of the builds which have many dependencies and subdependencies are firing multiple times.  Team city doesn't seem to detect that the same build trigger could fire multiple times due to the dependencies and ensure that the trigger only fires once.

There is also something else we would like to implement and were wondering what the best approach would be.  We have a project setup in Team city the purpose of which is to deploy all of our built components (war files) into our JBoss server(s).  Hanging off this project as a build dependency we have another project which runs our component level tests.  On successful completion of these tests we have another project set up to run Microsoft UIAutomation tests via fitnesse.  We were wondering if there was a way to prevent the first step in our process from occuring (the deployment) if any other builds in the sequence were still active.  What we have been finding is that we cannot prevent the build trigger which fires a deployment from occuring during the middle of our component tests or in the middle of the UIAutomation tests.  We suspect we may need to build another custom build trigger for this but would be grateful if anybody had any opinions.

Thanks very much in advance for any help

Keith

11 comments

Hello Keith,

Now we have more of a dependency graph or tree of projects as opposed to a dependency chain or sequence.  Several of our components depend on a number of other projects.  What we would like to implement is a build trigger which would wait for all of its dependencies to have been updated before launching a build.  We were wondering if this is possible out of the box or if we need to upgrade to 5.1 and create our own custom build trigger.  Specifically we are finding that some of the builds which have many dependencies and subdependencies are firing multiple times.  Team city doesn't seem to detect that the same build trigger could fire multiple times due to the dependencies and ensure that the trigger only fires once.


    I'd really appreciate if you provide an example of your dependency configuration and describe what's wrong with it (just name configurations A,B,C). That would allow us to understand better the problem you're having and to provide a possible solution.

There is also something else we would like to implement and were wondering what the best approach would be.  We have a project setup in Team city the purpose of which is to deploy all of our built components (war files) into our JBoss server(s).  Hanging off this project as a build dependency we have another project which runs our component level tests.  On successful completion of these tests we have another project set up to run Microsoft UIAutomation tests via fitnesse.  We were wondering if there was a way to prevent the first step in our process from occuring (the deployment) if any other builds in the sequence were still active.  What we have been finding is that we cannot prevent the build trigger which fires a deployment from occuring during the middle of our component tests or in the middle of the UIAutomation tests.  We suspect we may need to build another custom build trigger for this but would be grateful if anybody had any opinions.


   I suppose what you need here is a shared resource lock. We have a related issue in our tracker, and there is a workaround described in the issue comments.

   Best regards,
   KIR

0

Hi Kirill

Apologies for such a delay in responding, thanks for your help on this.  In answer to your first question here is an example.  Say we have a graph of dependencies like this.

    D
   / \
  B   \
/ \   \
A   E---G
\ /   /
  C   /
   \ /
    F

For example:-

G depends on D
G depends on E
G depends on F
D depends on B
E depends on B
E depends on C
F depends on C
B depends on A
C depends on A

What we are finding is that Teamcity does not understand the dependencies as a graph.  For example when someone changes A we would expect the following builds to occur.

B
C
D
E
F
G

What is actually happening is that we are getting something like the following sequence:-  I.e. Teamcity triggers a build per each route in the graph.

B
C
D
E
E
F
G
G
G

We are at the beginning of quite a complex project over here and we are evaluating if Teamcity can do what we need it to do.  We will have 8 teams of developers and they will all be working on components which will depend on components other teams are producing.  In the end we could have around 20 or 30 mini projects with dependencies between them.  Do you know if we would be able to put a complicated structure like this into Teamcity, or would we be misusing it.

Thanks for digging out that tracker.  That is exactly what we were looking for we have implemented the same sort of thing.  Do you think Teamcity will ever support this out of the box.  Also do you think that Teamcity will ever support the dependency graph stuff out of the box.

Many thanks for your help.  (Apologies for the mega slow response)

Keith

0

My picture didn't come out too good, If you cut and paste it into something else and change it to a fixed font it may be clearer.

0

Hello Keith,

   I suppose that for your structure you need the following settings:

   - for each of your dependencies, you should setup snapshot dependency betweeen configurations (for instance, you should add a snapshot dependency of build B on build A)
   - you should add _one_ VCS trigger to configuration G and enable option "Trigger on changes in snapshot dependencies". Other VCS triggers are not needed.
   - build dependency triggers are not needed with the setup like above
   When builds will be put to queue immediately after change detection, you'll see some duplicate configurations. But they should be eliminated over the time.

   At least this is how TeamCity is supposed to handle graphs of dependencies.

  Hope this helps,
  KIR

0

Kirill,

We are having a similar build issue, but we have another requirement: don't unnecessarily build configurations.  Extending Keith's example,

    D
   / \
  B   \
/ \   \
A   E---G
\ /   /
  C   /
   \ /
    F

the code built by each of our configurations comes from the same code repository, but from both shared and unique directory trees in that repository, e.g.

A - /a, /z
B - /b, /z
C - /c, /z
D - /d, /z
E - /e, /z
F - /f, /z
G - /g, /z

the simple case: a change to directory b must:

  • build only B, D, E and G, but
  • finish B before starting D and E
  • finish D and E before starting G


more complex case: a change to directory z must:

  • build all modules, but
  • finish A before starting B and C
  • finish B before starting D and E
  • finish C before starting E and F
  • finish D, E and F before starting G


How do I set this up in TeamCity?

Thanks,

Brian

0

Hello Brian,

  I believe in your case the setup should be quite the same as in the case I described above. It should work also, due to option "Trigger on changes in snapshot dependencies".

  Please let us know if it won't work.

  Regards,
  KIR

0

Kirill,

I figured out what is going on.  We use the Bazaar VCS.  Bazaar returns the same tip revision no matter where in the tree one queries it.  So, it appears to TeamCity that the entire tree has changed even if the change is limited to one part of the tree.  Subversion will return different tip revisions for different parts of the tree, so TeamCity can tell what has changed (and what hasn't) and build appropriately.

Barring a mechanism that will allow TeamCity to detect which Bazaar subtrees have changed, and which haven't, I think our only path forward is to break the working tree across multiple repositories.

Thanks,

Brian

0

We have a similar situation in our dependency tree except we don't have a convenient single top level build to set the VCS trigger on. We current have 43 build configurations in our "Build" project with overlapping dependencies.

What I did was create a "Master Build" build configuration that has a snapshot dependency on all of our builds. This "Master Build" has a VCS trigger that has the snapshot dependency enabled and is just a command line runner that runs a batch file that always exits with a 0 return code (it basically does nothing). All of the builds in the "Master Build" snapshot dependency have their own appropriate snapshot dependencies as needed for that particular build. We have no other triggers turned on.

When a VCS change is made in any build configuration (we use only one VCS root for ease of maintenance), it is detected by the "Master Build" and triggers the appropriate builds to happen in the correct order. Performance is better than I would have thought.

For all of our snapshot dependencies, we have "Do not run new build if there is a suitable one" and "Only use successful builds from suitable ones" checked, which should prevent uneeded builds from occuring. We use artifact depedencies to pass around dependency artifacts in a standard way. We also developed an app that uses the TeamCity Rest API to download the appropraite dependencies for our developers to use locally. This allows us to maintain one dependency list in TeamCity for our development environment, reducing errors and maintenance time.

For us the advantage is in the ease of maintaining build dependencies. Each build is only concerned with the builds it needs (via the snapshot dependencies). TeamCity sorts out the upstream dependencies for us. Also, maintaining the "Master Build" snapshot dependencies is a snap. Just delete all the dependencies and re-add them using a filter (all our builds have a standard nomenclature).Snapshot dependency also give you that nice roll up in the build results to see how the change affected other builds.

We tried using Build Dependency triggers, but they become stale, difficult to manage and prone to error in our experience. More importantly, they don't enforce any kind of build order.

Hope this helps.

0

This sounds like it's exactly what I'm looking for - except I must be missing something... How does TeamCity discover where a configuration resides in your CVS tree (that is, how does it know which folders it must be watching)?

From my understanding, when the VCS trigger hits your master build, all the snapshot deps are  triggered, and because of the nature of snapshot dependencies, all configurations will be rebuilt because TeamCity has no mean of checking which files/folders are used in the build process of a configuration.

How do you specify this?

0

We no longer do this now. We stopped doing it this way because performance was becoming very poor. We now only have VCS triggers on our top-level projects that having nothing dependant upon them.

The release of TeamCity v6.02 has dramatically increased performance of snapshot dependencies, so I believe we could go back to this now if we wanted too.

I'm not sure I completely understand your follow-up questions.

When you set up a build configuration, you associate it with a VCS root. That's how TeamCity knows where to get your source code and what changes to watch for. A VCS Trigger is separate from a VCS root. In this scenario, only the master build has a VCS trigger set to monitor snapshot dependency changes. You still have to set up your dependency tree using snapshot dependencies in each individual build configuration as well. The master build itself does not need to have a VCS root to have a VCS trigger.

When a checkin is made anywhere in the dependency tree, the master build will queue up all the build configurations it has a snapshot depenedency upon. Unless you set it to force build everytime (an option when you specify a snapshot dependency), it should only build the build configuration that had a VCS change. In turn, TeamCity will use the snapshot dependencies you specified to determine what other builds need to occur due to the change. This build resolution used to be very slow until the release of v6.02.

0

Actually I've answered my own question.

The problem was that all our subprojects reside in the same VCS root, and that to save some disk space I used to share both the VCS root and the VCS checkout rules, which made possible to have only one working copy.

I've changed that behavior. Now the VCS root is shared among the subprojetcs but the checkout rules are more restrictive, specific to one subproject, allowing TC to know which folder is which subproject. I've added also a VCS trigger on a meta-project that depends on all sub-projects.

Surprisingly I had issues with a structure with two end-of-chain subprojects, each of which with a VCS trigger: one of them didn't trigger with VCS changes higher in the chain. (Only one would trigger, the chain would build, and the second end-of-chain would stay with pendings, seemingly until a change was made to its own folder. To answer the oncoming question: Yes, I checked that "Trigger on changes in snapshot dependencies" was on.)

Thanks for answering, it's always nice to see a community commited to helping new members.

Best regards,
Jérôme Jolidon

0

Please sign in to leave a comment.