Is it possible to attach a file to a build I'm triggering?

Created September 24, 2015 08:40

I'm writing a plugin that triggers new builds (with the purpose of culprit-finding), using new BuildCustomizer().createPromotion().

I need to pass some data to each triggered build. Currently I'm doing that by specifying parameters (buildCustomizer.setParameters()). However, the data is of arbitrary length and I'd prefer to attach a file to the triggered build. Is that possible somehow?

If files are not an option, and I must use parameters: how can the triggered build handle the parameter safely? Suppose the build needs to pass this to a script foo as a command-line parameter. I can't simply call foo -a "%my.param%" because %my.param% is potentially too long for a command line, and/or contains new lines or quotes.

Any suggestion?

7 comments

QA Platform

Created September 30, 2015 12:10

I'm guessing it's not possible to attach files to queued builds.

In the end I solved it as follows:

when the plugin is adding the build to the queue, it sets the data into a custom parameter of the queued build
later, when the build starts, the plugin kicks in again; this time it's an agent-side plugin which takes the parameter value and creates a file in the build temporary directory (the sourcesUpdated() event - it can't be the buildStarted() event because that happens before the build temporary directory is cleaned up)

This way, the data is only every handled by Java (by the server-side plugin in #1, by the agent-side plugin in #2), never passed around on the command line. The build steps of the build can also operate on the data in the file - they never have to receive the data as a command-line parameter.

Vladislav Rassokhin

Created October 01, 2015 20:09

Hi Sam,

There'is no direct method to 'attach' file to build from server side, afaik.
What do you try to achive? Maybe there's another approach to do that.

Can't that file be generated on agent side?

QA Platform

Created October 02, 2015 08:23

Hi Vladislav,

Yes, that's what I ended up doing: I am now creating the file on agent side.

Here's what I'm trying to achieve: I have a plugin that triggers some builds. (The purpose is to find who in a range of changes broke the build. But that's not important.) The plugin needs to pass some structured data (a list of failures for which we're trying to find the culprit) down to the builds it triggers. How should it do so?

It seems the only reasonable way to pass data to queued builds is to set parameters. So my plugin sets a parameter for every build it queues. Given that this is structured data, the parameter is a (potentially long) JSON string.

So far so good. The builds that are thus triggered receive the necessary data in that parameter. But, how can they read it? Suppose each such build has a step to process that JSON string, and do something based on its contents. That step could be a command-line runner, e.g. calling something like ProcessMyParameter.exe -value "%my.parameter.containing.json%". Maybe you're starting to see the problem:

the JSON string is potentially too long for a command-line (the command-line could overflow its length limit)
the JSON string may contain double-quotes (in which case the quoting breaks)
the JSON string could contain new-lines (in which case the command-line will be broken prematurely)

So, it would be much nicer to be able to put the JSON string in a file. Then my build could call ProcessMyParameter.exe -file my_parameter_file.json instead and avoid all the problems above.

But then... how do you put that JSON string into my_parameter_file.json? You can't have a command that runs echo "%my.parameter.containing.json%"> my_parameter_file.json - that would suffer the same problems.

I ended up creating an agent-side plugin which puts the JSON parameter and puts it into a JSON file. That way, the JSON string makes it into a file without ever having to be exposed on the command-line.

Can you think of a better way to do this?

Thanks,
Sam

Vladislav Rassokhin

Created October 02, 2015 14:23

Hi Sam,

Your idea looks good, our 'Run railed tests first' feature works almost the same way: generated file in per-build temp directory with new-line separated list of tests + build parameter with absolute path to that generated file.
So you could use same idea:
1. Generate build parameter 'XList' on server
2. On agent save 'XList' into 'YFile'. Add 'YFilePath' parameter with path to 'YFile'

That have some benefits:
On server UI you will see what tests are requested to run to detect value (see 'XList' value on build parameters tab)
Script/Runner can use either 'XList' or 'YFilePath' parameter. Second is safe in terms of arguments lenght, quotes, new-lines.

> It seems the only reasonable way to pass data to queued builds is to set parameters.
That's parameters main purpose.

Regards,
Vladislav

QA Platform

Created October 07, 2015 10:10

Thanks. It seems we have ended up doing the same thing.

One more question: at which stage does the agent-side plugin write XList to YFile? I've tried the following events in AgentLifeCycleListener:

buildStarted() - this is intuitively the right stage, but unfortunately it's called before the cleaning up of buildTmp, so if I create YFile here, it will be wiped by TeamCity a split second later (in jetbrains.buildServer.agent.impl.buildStages.startStages.CleanBuildTempDirectoryStage)
sourcesUpdated() - this is the next candidate, but this event is never triggered for builds which don't perform checkout
beforeRunnerStart() - this is also an option, but (1) it's triggered for each build step, which is rather sub-optimal, and (2) there could theoretically be builds without build steps, and then it would never trigger - although I admit that's a rather convoluted case

Thanks,
Sam

Vladislav Rassokhin

Created October 07, 2015 18:27

I could say how our 'run failed tests first' logic ('jetbrains.buildServer.agent.impl.buildFiles.RecentlyFailedTestsFileWriter'):

On `buildStarted` property with genereted path resitered using `build.addSharedSystemProperty`
On `beforeRunnerStart` wile written into file (file path get from that shared system property) and also rewrite action registered as `BuildRunnerContextEx#onParametersUpdated` (you may not neet that rewrite part)

Actually our code is a bit complicated, but that's essential parts. So your case could be implemented same way.

QA Platform

Created October 08, 2015 09:05

OK, thanks. We use beforeRunnerStart() too. (I had made a mistake above and wrote buildRunnerStarted() but I've updated that now.)

Please sign in to leave a comment.