Using artifact dependencies as an environment/system variable?

As part of a hierarchical build project I'm working on, I'd like to be able to poll a configuration's artifact dependencies in order to build a file listing what those dependencies are.  To better illustrate my issue, assume the following:

Configuration A, B, and C output the artifacts AFile, BFile, and CFile respectively.
Configuration Z has a dependency on all three of these artifacts.

I want a way for the build script run by Configuration Z to reference AFile, BFile, and CFile without having to manually input the file names into its environment/system variables.  Is there a way to do this?

5 comments

Dave,

If I understand you right, you have TeamCity artifact dependencies configured and need to proces the downloaded files inside the build script of Z. The easiest way to do this seems to download the artifact dependencies into distinct directories and then process all the files in these directories inside the build script.

I am afraid that it is not currently possible to obtain the downloaded files inside the build. This is avaialble in the API and can be seen on the build's Dependencies tab. This functionality can be added into the REST API plugin in the future, if really necessary.

0

Unfortunately, that was not what I was asking; but does provide at least one solution to my problem.

The question is if there is a way to extract the information about a configuration such that the build script can use it.  In other words, since Configuration Z has three artifact dependencies, is there a variable containing a list of said dependencies.  If we continue the current example, what I want is a variable that, when printed, would list AFile, BFile, and CFile.  To extend upon this, there would also be a variable listing the configurations these dependencies come from; when printed would list Configuration A, B, and C.

The reason I'm asking for such variables is because placing them myself, or parsing them in the manner you've already mentioned, should be considered redundant work.  I should have a way to simply extract this information from TeamCity.  Other than making a plug-in, do these variables already exist, or is there an easier way of extracting such information?

0

Dave,

For the time being there is no variable that holds the build configuration dependency settings.
For now this does not seem a very common use case, but you can file an issue so that other users can vote.

BTW, can you describe why do you need this?

0

yaegor wrote:

BTW, can you describe why do you need this?


Certainly!  I probably should have explained earlier anyway.

Our company is moving towards an environment where our machines are deployed-to entirely from images.  Using programmatically generated images, we create machine images using sets of debian packages.  What I want to do with TeamCity is allow for a configuration that is dependent upon the debian package configuration's completion in order to build a final image (or images) that make up a project.  The script which builds the final image needs to know what debian packages are required, which it should be able to grab from its TeamCity configuration.

Upon further thinking, artifacts actually make this whole situation worse, so let's take them out of the equation.  Here's a quick explanation of a set up we need.

Setup:
Deb-maker1, Deb-maker2, and Deb-maker3 are three separate TC configurations.  They run when changes are made - they complete by uploading a debian package ("Debian1.deb" "Debian2.deb" and "Debian3.deb" respectively) to S3.
Img-maker1 is a TC configuration that has a build trigger dependency on Deb-maker1, Deb-maker2, and Deb-maker3.  When it completes, it uploads "Image1.img" to S3.
Assume Deb-maker1/2/3 all successfully built a debian at least once in the past.

Process:
When Deb-maker2 is triggered by a change, it builds and tests its code then uploads Debian2.deb to S3.  Img-maker1 notices Deb-maker2 just succeeded and kicks off its script. (Here is the important part)  The script should use its TeamCity configuration's knowledge of its build trigger dependencies in order to retrieve the 3 Debian packages it requires from S3 in order to build its image. By having this information, the script builds the image and uploads it to S3, thus completing Img-maker1's task on TeamCity.

Problem:
Unfortunately, because the script currently has no way of polling TeamCity's information to know what its dependencies are, I have to do some redundant work-around to retrieve that information; which leads to bad mistakes about what the image needed to build and unneeded complexity in our configurations.  An image can (and will) require anywhere from a small set of debian packages to dozens, which makes the redundant work that much more labor intensive.

Looking through the plug-in API it seems like there's no real way to work around this problem other than hoping that you guys can expose this information to agents or allow polling of the server by the agent's script to query what the dependencies are.  I'll be sure to add this as an issue, but if you think there's a better way to handle this than simply allowing the trigger dependencies as a system variable, please let me know and modify the issue.

Edit: The issue can be found at http://youtrack.jetbrains.net/issue/TW-11551?projectKey=TW

0

Dave,

Thank you very much for the detailed description.

I'll try to reformulate the task:
image builder uses several packages that are produced by Teamcity builds
The image/packages correspondence information should be used in:
1. triggering image builder build upon update of any of the packages
2. actually downloading the packages (from S3) to build the image

So the main task is to determine what will serve as the primary storage of information for the packages/images relations.
It seems not quite appropriate to use Teamcity dependency triggers to store the information, since triggering is a quite dumb thing: it just triggers one build on another build finish and is not designed to store any kind of dependency information.
Moreover, if in the future you will want to add dependency trigger on non-package build, you will need to filter that out to determine packages.

There might be ways of storing the information outside of TeamCity and just ensuring the necessary image build is triggered when there is an update in one of the necessary packages (e.g. the package file is really updated on S3 -much like TW-9851, but for S3).

I'll describe a way I'd try to do it in TeamCity nowadays:
- make each package build produce an artifact - a file with package description (e.g. id, name, path that the real package file was published to)
- in TeamCity for the image build configure artifact dependencies (this will actually define the necessary packages) and also dependency triggers. Defining both artifact dependencies and dependency triggers  requires some duplication, but in return you have ability to depend on a package and do not rebuild image on its change, if that can become necessary. Also some day TW-11553 will also help here. The artifact dependencies should put all the descriptor files into a single directory, and use the option to clean the directory beforehand (to clean after the previous build).
- in the image creation build, grab all the descriptor files from the directory and use the information to download the packages from S3.

0

Please sign in to leave a comment.