Extracting build dependencies

Hi,

What is the best way to extract build dependencies from teamcity in an automated fashion?  Having a build type id what is the best way to get what builds the build type is dependent on?  I don't think this information is available through the rest api.  I'm wondering if it is less work to just scrape the UI or write a plugin to get this information?  Any suggestions would greatly be appreciated.

Thanks,
Jay

14 comments
Comment actions Permalink

I can't claim that my way is the "best" way, but it works for us...

Usually, a dependent build will require build artifacts, and I extract the build information from the build artifact. In fact, I have most builds produce a standard result artifact from which I can then retrieve all of the inforation I wish to get, including the download locations etc (since I don't actually store large artifacts on the TC master).

http://automated-builds-secondlife-com.s3.amazonaws.com/hg/repo/snowstorm_viewer-development/rev/230223/index.html

0
Comment actions Permalink

So on the build you are dependent on you actually create an artifact in that builder and then pick up that artifact on the build that has the build dependency set?  Can you put the buildtypeid in the artifact?

Otherwise I was thinking of extending the rest api and maybe put on the buildtype rest url the dependent buildtype ids.

0
Comment actions Permalink

To get the build type id, I pass in an environment variable set to                                       %system.teamcity.buildType.id, and yes, that gets plunked into the url pointing to the TeamCity task that created the artifact.

0
Comment actions Permalink

Hi Jason

REST API allows to obtain list of snapshot dependencies in dependency-build property.

There is no info for artifact dependencies yet, feel free to create a request in our tracker.
It would be helpful if you provide some examples there, how are you going to use this data.

Thanks
Michael

0
Comment actions Permalink

Thanks, I created a ticket here:

http://youtrack.jetbrains.net/issue/TW-16946

For now I have extended the rest api but would prefer it be part of the product so I don't need to keep building a new version after releases.

Thanks,
Jay

0
Comment actions Permalink

Christian,

The page at the link you gave looks interesting. Is it generated from TeamCity somehow?
Can you describe how and why a bit?

0
Comment actions Permalink

Ok, long story..

We use S3 as our actual build artifact store. The layout is documented at

https://wiki.secondlife.com/wiki/Automated_Build_System#Uploading_Build_Results_to_S3

That page also describes roughly how our build system works - but it is slightly out of date... especially the contents of the public source code repo for the build scripts.

The only "proper" TeamCity build artifact I create is a little html file containing a viewer side redirect to the actual build results page living on S3. The redirect works like this:

<HTML>
<HEAD>
<META HTTP-EQUIV="Refresh" CONTENT="0; URL=http://automated-builds-secondlife-com.s3.amazonaws.com/hg/repo/merov_viewer-development-features/rev/230469/index.html">
</HEAD>
</HTML>


The nice thing is that anyone clicking on the build artifact will end up on the results page, and the format is simple enough to be trivially parsable.

When a build has an artifact dependency to another build, the build script will read the viewer side redirect, extract the S3 URL and then either already know where to look for the actual build artifact, or will be able to retrieve it by parsing the index.json file stored next to the index.html build result page on S3.

That way, all my artifact dependencies look exactly the same from TeamCity's point of view, and explaining how to add them is easy (and *should* be trivially automatable, GRRR :)  ) they all contain the exact same parameters - the only thing that matters is which build config you actually pull them from.

The changeset info, changes since last good build, summary info and such are all generated independently of Teamcity, mainly so that people can reproduce our build environment outside of TeamCity, but also because TeamCity cannot have all the information we need. For example, my "changes since last good build" include not only the changes in the VCS roots used for the build, and not even all the changes of all the dependent builds as expressed by the TC artifact dependencies, but of all the changes of all dependencies as long as they have been built by this build process, be it via TeamCity or any other means.

0
Comment actions Permalink

Christian,

Thank you for the description. Looks really interesting.

I wonder what TeamCity features can help in your or alike tasks...
e.g. something like TW-1558 (External artifact publishing) or TW-2576 (Include changes from changed artifact dependencies in the build's changes).
If you will ever want to reduce the number of the parts in the system, there seems to be a way to write a TeamCity plugin which will make most of the required data available inside TeamCity itself.
However, that's probbaly not the goal, just one of the further development ways.

Anyway, if you have feedback on TeamCity, suggestinos, etc. make sure you post them here in the forum or in our tracker.

0
Comment actions Permalink

I think there needs to be a cleaner separation of concerns. In other words, TeamCity has a core competency: organizing the scheduling and running of builds. Tracking the changes that go into a build, or even features like remote artifact storage are good and nice, but not part of the core competency. In fact, I'd argue that even if they were there, I would be reluctant to rely on them, because if I do, I am suddenly completely dependent on TeamCity - right now, I am not. Also, there are so many variants and special cases between all of your users that attempting to integrate them into a single feature would either be extremely complicated, or unsatisying to many users.

I think TeamCity needs to make all components of the data used to schedule a build available, so that build engineers can write whatever mechanism they want to complete the build data, but I'm not sure it is a good use of resources for TeamCity to attempt to do that for them. There is a big risk of creating someting like Jira, which tries to be everything to everybody :)

Now there -are- features which I believe do belong in your core competency which currently are not addressed: mainly I'm thinking of the situation where a build has multiple build completion triggers. I would like to be able to say: "If a build competion trigger fires, please check if all the other build completion triggers have a build running, and don't run the build yet. Only if a build completion trigger fires and there are no more other builds referenced by other build completion triggers, then run it.

The use case for us is an expensive aggregation build triggered by cheap packaging builds (specifically, debian packages trigger a debian image build, uploaded to EC2 as an AMI and tested). Often, multiple packages get updated simultaniously, and whichever package build finishes first triggers an image build, which takes about 30 minutes to complete. Those 30 minutes are wasted, because the image is useless without the other package updates. Sometimes the image build will fail, which at least saves some time, but create bad stats, as that failure is really due to the inability to schedule a correct build instead of some flaw in the process.

Alternatively, a "quiet period" for build completion triggers might also work....

0
Comment actions Permalink

Christian,

I think TeamCity needs to make all components of the data used to schedule a build available, so that build engineers can write whatever mechanism they want to complete the build data, but I'm not sure it is a good use of resources for TeamCity to attempt to do that for them.


Can you please elaborate on this? Any specific examples or use cases to cover?


The part with build completion triggers synchronization can probbaly be addressed already by using snapshot dependencies: a build that snapshot-depends on other will not start until all dependencies are ready.
The Snapshot Dependency notion is a bit complex, but it is a "smart" depenency", unlike finish build trigger one. Finish build trigger is a simple and non-intelligent one and so far we are not sure it makes sense to make it smarter.
We will probbaly try to make snapshot dependnecy easier to understand/use, however.

0
Comment actions Permalink

As to the first part, I am fairly happy - since I use agent side checkouts, I can depend on the VCS and the dependencies as expressed in the build system stored on version controlled files gives me all I need. Essentially, I use artifact dependencies to expand the "search space" where dependencies can be resolved, but the actual dependencies are extracted from the build itself - i.e. I don't trust second-hand data :)

I will experiment with snapshot dependencies some more, but I don't think it solves the problem. I don't want to have to trigger all package builds before an image is built, I only want all the currently running package builds to complete before an image build is attempted. I'll see if this is expressible somehow within the options provided...

0
Comment actions Permalink

Snapshot dependency has Do not run new build if there is a suitable one option.
It allow to use artifacts from last successful build, and do not triger it every time.

0
Comment actions Permalink

So how does the build with the snapshot dependencies get triggered?

0
Comment actions Permalink

You can configure Build Trigger to depend on the same Build and trigger on a successful build which is what we do.

0

Please sign in to leave a comment.