Problem coordinating builds of layered projects

We have a number of Maven projects for which we're using TeamCity.  We would like to achieve a sequence of hierarchical nature, for instance:

Dependency structure:
- project A / build configuration for A in TeamCity / has a VCS trigger for A only
- project B / build configuration for B in TeamCity / has a VCS trigger for B only
- project B depends on project A

Build use-cases:
1) A changes. Desired effect: build A, then rebuild B.
2) B changes. Desired effect: build B. Use latest existing build results for A.
3) A and B change simultaneously in the same check-in.  Desired effect: build A, then build B.

Problem statement:
We can make use-case 1 work no problem.
We can make use-case 2 work no problem, with additional trigger on TeamCity build of A.
Our issue is mostly use-case 3. How do we ensure that A is built first then B is built upon a simultaneous check-in?

9 comments
Comment actions Permalink

I work with Dmitry and have also been looking at the problem. We are aware of https://youtrack.jetbrains.com/issue/TW-4107 but I'm not sure it would solve the issue we face, since it seems to be a race condition of whether or not project A notices the VCS change before Project B. I have tried configuring Project A as a Snapshot dependency of B, but we still get out of order builds unless I specify that it runs every time. (Uncheck all boxes in dependency configuration pop-up). This fixes Use Case 3, but breaks Use Case 2 where we only want B to build. (note that in fact we also have C, D, E depending on A and more than one "A" ... but we simplified things for this dicussion)

Basically we want Build B to look at it's dendencies (Build A) and force them to poll for VCS (or other trigger) changes and queue themselves before Build B queues it self.

We're using Team City 8.0.5 (build 27692)

0
Comment actions Permalink

Hi Dmitry and Gus,

If B snapshot depends on A, you can configure one VCS trigger in B build configuration with option "Trigger a build on changes in snapshot dependencies" option enabled. No VCS trigger is required for A build configuration. In this case:

1. A changes. The whole build chain is rebuilt.
2. B changes. Build B added to the queue. If suitable build A is found it is used.

3. A and B change simultaneously in the same check-in. The whole build chain is rebuilt once.
See also an example at the Build Dependencies section. Also if there are similar builds in the queue they will be optimized.

BTW, I would recommend to upgrade to the latest TeamCity version (9.1.3 as of now) to get access to all the recent bug fixes and lots of new features.

0
Comment actions Permalink

HI Alina,

I'm still struggling with this. I tried your suggestion and it didn't seem to work. I convinced the team to upgrade to 9.1.3 hoping that your suggestion would work there but it still does not. Let me describe my configuration in some more detail. I'm wondering if VCS roots play a role here.... Things just don't seem to work as you describe them.

In your example B depends upon A. In my system B is called "Ingestion Service" and A is called "Pipeline". They are physically stored i a single SVN repository for historical reasons, but pipeline is a distinct component, and the VCS configurations are somewhat different.

  • Pipeline has been configured with a VCS root that views a subset of the repository (starting from one level down from the root).
  • Pipeline does not have any build triggers (as suggested)
  • Ingestion Service customer specific and lives 2 directories down in a sibling to the directory in which Pipeline builds
  • Ingestion Service has a separate VCS root that starts at the top and then uses checkout rules to ignore everything else (including pipeline with -:/pipeline ).
  • Ingestion Service has several snapshot dependencies including Pipeline. Snapshot dependency configuration is as follows (for all):


Do not run new build if there is a suitable one

On failed dependency: make build failed to start

On failed to start/canceled dependency: make build failed to start

  • Ingestion Service has a VCS trigger as follows:

Triggers a build on changes in snapshot dependencies
Quiet period: 60 seconds (default)
VCS trigger rules:
+:root=InjestSubversion:customers/customer-foo/**
Triggers one build per each VCS check-in (include several check-ins in build if they are from the same committer)

When I check code in I get the following:

  1. Check a comment into the pom at /customers/customer-foo/pom.xml
    • Pipeline is not built (correct)
    • Igestion Service is not built (correct)
  2. Check in a comment into  /pipeline/pom.xml & wait 5 minutes
    • Igestion Service is not built (incorrect)
    • Pipeline Service is not built but lists my checkin as a pending change (incorrect)
  3. Delete both comments and check them in simultaneously.
    • Pipeline is built (correct), but the change from the simultaneous checkin remains pending (incorrect) (the previous checkin from step 2 was included)
    • then Ingestion Servicce is built (correct) without the chage checked in at the same time (incorrect and build might fail when it should not)


I tried removing the -:/pipeline checkout rule from Ingestion Service, but that didn't help

This was all done on TeamCity Professional 9.1.3 (build 37176)

0
Comment actions Permalink

I saw your request in tracker: https://youtrack.jetbrains.com/issue/TW-43065 but decided to reply here, because I think what you described is not a bug. First of all in your first message and in the summary of the bug report you did not mention trigger rules, while this is an essential part of your configuration. The way how you described the problem, it really looks like a bug in TeamCity. However, we would not be able to reproduce it because you omiited part with triggering rules.

Speaking about triggering rules, you mentioned that you have this in VCS trigger of "Ingestion Service":
VCS trigger rules:
+:root=InjestSubversion:customers/customer-foo/**

Note that since you enabled option "Trigger on changes in dependencies", triggering rules are applied to all of the changes: own changes of build configuration and those which came from dependencies.

I do not see anywhere in your message what InjestSubversion is. So I have to assume this is the id of VCS root from "Ingestion Service" build configuration. With this setup you told TeamCity: trigger only on changes under customers/customer-foo/ directory made in VCS root with ID: InjestSubversion. Is this what you want?
Judging from your later comments, probably not. You mentioned that checkin with file /pipeline/pom.xml did not trigger a build, and this is correct because with your configuration of triggering rules it should not.

0
Comment actions Permalink

Hi Pavel,

Thanks for replying. Some of the initial description here is what we were initially trying, we modified our configuration based on the response here. I initially was impatient and mesed up on the bug report and mixed the two. Sorry about that. The bug relates to the modified configuration. I will now make a fresh attempt to describe a 2 build segment of our overall configuration (we have 20 builds) that demonstrates the problem we have at multiple levels. This includes our attempts to incorporate the above advice, and excludes our initial configuration, or other attempted work arounds (some of which I describe in the bug report).

Here's the full details on the VCS setup in question (customer name replaced etc). The overall pattern is B depends upon A.

  • One physical subversion repository. (denoted by /trunk/ below) - but this should be irrelevant? what if they were separate repositories on separate svn servers? That should work too...
  • Project A has a dedicated subversion root (lets call it SubversionRootA)
  • Project A has VCS config like:
    • checkout from /trunk/platform/projectA
    • No checkout rules.
  • Project A runs a maven build from Path to POM: pom.xml
  • Project A has no build triggers (as suggested above)
  • Project B has a dedicated subversion root (lets call it SubversionRootB).
  • Project B has a VCS config like:
    • checkout from /trunk/platform  
    • Checkout rules like:
      • -:/projectA
        -:/projectC
        -:/projectD
        -:/projectE
        -:/projectF
        -:/projectG
        -:/projectH
        -:/projectJ
  • Project B runs a maven build from Path to POM: customers/customer-foo/projectB/pom.xml
  • Project B has a single VCS build trigger:
    • Triggers a build on changes in snapshot dependencies
      Quiet period: 60 seconds (default)
      VCS trigger rules:
      +:root=SubversionRootB:customers/customer-foo/**
      Triggers one build per each VCS check-in (include several check-ins in build if they are from the same committer)
  • Project B has snapshot dependency on A configured as suggested above:
    • Do not run new build if there is a suitable one
      On failed dependency: make build failed to start
      On failed to start/canceled dependency: make build failed to start


In this configuration changes to A do not trigger a build of B, despite the trigger rules for B specifying a build on changes in snapshot dependencies.

I DO want a trigger of +:root=SubversionRootB:customers/customer-foo/** because I don't want to trigger on changes to customer-bar's code. Nothing in the documentation says anything about the trigger rules from B being applied to A? That doesn't even make sense to me, since in other setups A might come from an entirely separate Git or Mercurial or whatever... why would trigger rules for B have anything to do with the detection of changes in A?

From your documentation (mentions nothing about trigger rules):

Trigger on changes in snapshot dependencies

Another option that alters triggering behavior within a build chain you can find in the VCS build trigger options. It allows to trigger the whole build chain even if changes are detected in some further build configuration, not in the root.
Let's take a build chain from the first example:

Pack setup
--depends on-->
Tests
--depends on-->
Compile
.
Normally, the whole build chain is triggered when TeamCity detects changes in
Pack setup
, changes in
Compile
do not trigger the whole chain - only
Compile
is run. If you want the whole chain to be triggered on VCS change in
Compile
, add a VCS trigger with "Trigger on changes in snapshot dependencies" to your
Pack setup
configuration.
This won't change the order in which builds are executed in any way.  This will only trigger the whole build chain, if there's a change in any  of snapshot dependencies.

0
Comment actions Permalink

As a side note, after reading and attempting the suggestion from alina I've wondered what one should do if one wanted to ignore some types of changes in the snapshot. Imagine a /docs directory under customer-foo with customized documentaiton for that customer's custom, code for example.

This is why I initially thought a VCS trigger would be required on the snapshot... so that there would be some sort of definition of what qualified as a "change".

0
Comment actions Permalink

TW-43065 has become confusing because of my initial poor description. Do you want me to open a new ticket with the above (hopefully clearer) description of the problem?

0
Comment actions Permalink

The fact that documentation does not mention how triggering rules are applied to changes from snapshot dependencies is a bug: https://youtrack.jetbrains.com/issue/TW-43102

Note that VCS trigger works like a filter for all of the pending changes. Trigger itself does not detect any changes, it just waits for their appearance in database. After the changes appearance, trigger filters them in order to find commits which can trigger a build. And at this point it does not distinguish whether commit came from own build configuration repository or some dependency. This is done intentionally because otherwise it is not clear how one would be able to suppress triggering on some changes coming from dependencies.

0
Comment actions Permalink

Hi Pavel,

Thanks for clarifying how the present system works and opening the documentation bug. Since you've pointed out that the present design is functioning as intended, here's a feature request to make the builds easier to maintain, and more intuitive overall in the future.

https://youtrack.jetbrains.com/issue/TW-43117

Thanks for your help with this,
Gus

0

Please sign in to leave a comment.