Snapshot dependency fetching week-old builds - anyone know why?


I'm seeing some issues in our TeamCity (8.1, 29879) setup which don't seem to correspond with the intended behavior, and I'd like to know if anyone else has had the same problem or knows what the cause is. In short, what happens is that a snapshot dependency we have set up will resolve sources/builds which are NOT from the same point in time, causing incompatability issues in our system.

Our setup looks something like this:

  • We have some 10 modules, each with their own VCS root and TeamCity project, and one build configuration each. The configurations will run some local tests and produce one executable artifact each. They will trigger on VCS check-ins. Let's call these build configurations A1, A2, A3, and so on.
  • We have a build configuration, B, which has artifact and snapshot dependencies on the A configurations. B will start the executables and run a system test. B has a "Finish Build Trigger" for each of A1, A2, etc.

Now here's the strange part:

  • When someone does a code check-in to e.g. the module corresponding to A10, a build will start. If the build is successful, B will be triggered.
    • With "Do not run new build if there is a suitable one" turned off, A1-A9 will be queued, get built, and used as inputs for B (correct behavior).
    • With "Do not run new build if there is a suitable one" turned on, semi-random builds from A1-A9 will be used. We've seen cases where e.g. a week old build of A1 will be used, even though there are several successful builds that are newer (wrong behavior).
  • When I trigger B manually:
    • With "Do not run new build if there is a suitable one" turned off, the behavior is the same as above (correct behavior).
    • With "Do not run new build if there is a suitable one" turned on, all the newest builds from A1-A9 will be used (correct behavior).

My understanding of snapshot dependencies is that, when the build configurations your are depending on have different VCS roots, TeamCity will find the newest check in for each configuration that is not newer than the one that started the chain, and build those check-ins (or used equivivalent, finished builds if "Do not run new build if there is a suitable one" is turned on), but this does not seem to be the case. I also noted that many (but not all) build logs have the line "Skip checking for changes - changes are already collected" at the top of the log, which I'm not sure how to interpret.

Any help or ideas are welcome!

Comment actions Permalink

I'm seeing much the same problem. I've got no idea where team city is even getting the old snapshot from. The version in Artifactory is correct, the version in the local repo on the machine is correct.

When I run mvn manually from the command line in the buildagent work directory the correct (most recent) snapshot is used.

When teamcity runs the build an old snapshot is retreived from somewhere and everything is broken.

Pretty game breaking problem....

Comment actions Permalink

When you have several builds in chain and have "Do not run new build if there is a suitable one", then TC can skip configurations that already have all necessary changes.

"Skip checking for changes - changes are already collected" is a normal message, because changes are collected before the whole chain starts.

You can look into teamcity-server.log for messages like this: "Build queue optimizer substituted 1 with already started builds" and if you enabled debug logging, you will have more insight into why TC prefers an old build versus a new one.

Comment actions Permalink

Are you talking about having dependencies set in the build configuration?

I don't have any of those.

I only have dependencies declared in the maven pom.xml and the maven dependency trigger set.

We also have only 1 build agent.

The problem has now mysteriously dissappeared making further diagnosis difficult. I feel like I'm sitting on a ticking time bomb with this...

Comment actions Permalink

Please read the following to understand how it work:

Probably, it will make things clearer.

Comment actions Permalink

I could be missing something, but my understanding of that is that a build may kick off with an old snapshot and then a new snapshot become available, meaning the build has used a snapshot that is not the latest version.

I don't see how this can apply when

1.all builds are complete
2. nothing is in the build queue
3. both artifactory and the local repo contain the correct snapshot version
4. I queue the build manually

Obviously I'm missing something, but I don't know what that might be.

Comment actions Permalink

You probably need to check option "Do not trigger a build if currently running builds can produce snapshot dependencies" for maven snapshot and artifact dependency triggers.

Comment actions Permalink

I've done exactly that, but I'm still concerned. I don't like blindly choosing options and hoping my problems will go away.

I'd much prefer to understand the mechanism that could result in an old snapshot being used. If there are no builds running, no builds queued and the correct snapshot is in artifactory, why would team city use an old snapshot when I manually kick off a build? Is it remembering the snapshot version it had for the last build and not bothering to check artifactory or the local repo for updates?

BTW : I appreciate you putting time into trying to help me.



Please sign in to leave a comment.