Groovy plugin script issue after upgrade to 6.5.1 from 5.1

Additional detail. This is what I'm trying to accomplish and which used to work before.

I need to generate properties when the build is running, and since TC does not support that I am using service messages to publich a build number which encodes the values of the properties I need to populate, then using a Groovy script I'm parsing the properties and adding two new properties to the build.
Now it appears that those properties are generated correctly (appear in the dump using system.teamcity.debug.dump.parameters) when a build A already has a build number with the right format at start time (it's carried from a dependent build, for example), but if I create a build B which depends on A, those two properties don't appear with the dep. prefix.

Do you see generated properties in 'Build Parameters' tab?

Starting from TeamCity 6.5, build agent is able to report all build parameters back to the server. Those parameters are returned from the build when constructing dep.* parameters.
TeamCity will not call any parameters providers for finished builds.

It is now possible to set parameters from build on agent side. Please take a look at
http://youtrack.jetbrains.net/issue/TW-15503

Eugene, interesting feature that I asked for some time ago but now I need to make it work with the current mechanism, which with 6.5 seems to be broken.
I do not see the properties in the "Build Parameters" tab of the build config which is supposed to generate those properties, but I'm not sure it's something I should see since I don't see any properties except for those explicitly defined via the web UI.

Further explanation of the issue.

If using the Groovy plugin I define a property in the same way but without the regex stuff going on it is populated correctly and I can see it with the dep. prefix in a dependent build, the problem seems to occur with that logic in parsing the build number, which apparently no longer runs at the right time and thus does not generate my two additional properties.

Simone,

So the initial target is to provide the properties to a finished build so that builds that depend on it can use them?

With 6.5.1 you can publish a property from a build so that it will become available to other builds via dep. property.
Publishing the property is available via a service message.

As to the issue with your customization of GroovyPlug. How do you call addProductVersionAndAssemblyVersionParameters ?

Can you share all the customized sources?

> Also, I can't seem to find a way to enable logging for this plugin, any advice?

Attached is a log4j config to log all debug-level messages into teamcity-groovyPlug.log log file.
You can place it under .BuildServer\config\_logging and select from web UI under Server Configuration/Diagnostics

Yes, that is the aim. I am generating the build number on the fly using service messages using my build script, then that number has to be parsed using the Groovy plugin and those two properties containing parts of the build number have to be published so that they are available to dependent builds. I realize now it is possible to define those properties directly during the build, but in 5.1 it was not possible and I relied on this workaround that needs to work fine in 6.5 as well to perform an in-place upgrade.

I attached the whole groovy script, which is the original one with some additions, including this one.

Thanks for the log config file.

Simone,


OK, I figured out what is wrong here. We indeed changed the behavior in 6.5 that actually prevents plugins from sdding properties for finished builds.

I've filed an issue on that: TW-17466 and Eugene, should describe current workaround there.

Thanks Yegor, but its not working yet unfortunately. Here are more details:

I am running build A and during the build process I am generating a build number that I push to TC via service messages in the form 1.2.3.4 (5.6.7.8).
Then runs build B which runs tests on the artifacts produced by A but does not depend on the properties generated by the goovy script. If I enable parameter dumping for build B using system.teamcity.debug.dump.parameters I can see that the parameter dep.btA.build.number.productVersion is indeed there (dumped at the beginning of the build log), so it would appear that your fix is working.
Finally I have build C which depends on A and B and uses the dep.btA.build.number.productVersion, but TC says there are no compatible agents due to missing dep.btA.build.number.productVersion property.

Note that I added those two properties as explained by Eugene here http://youtrack.jetbrains.net/issue/TW-17466 to all the agents.

One more detail. Build B dumps the dep.btA.build.number.productVersion correctly to the build log as I said before, but as soon as I add a property to build B with value %dep.btA.build.number.productVersion% then the behavior is the same as for build C, that is, the dependency is not satisfied due to missing property and thus the build no longer starts.
So it appears the property is there when the build starts, but as soon as you add it as a precondition TC does not consider it satisfied.

I just postted a fresh workaround for you at
http://youtrack.jetbrains.net/issue/TW-17466#comment=27-232724

The idea is to add all dep.bt*.productVersion and dep.bt*.assemblyVersion properties in the build that consumes, but not in the builds of bt*.
In your groovy code you simply can search for all dep.bt*.build.number and to the same  loging as you does but adding properties with same dep.bt*. prefix.

What properties do you see in the 'Parameters' tab of a build?

TeamCity collects parameters from all dependent builds (transitively).
Please check you do not see there any build that was started before you diabled properties upload from agent

In builds B and C I see no parameters in addition to the ones defined explicitly in the build configuration.

The idea was the following.

Say Z depends on X and Y.

Your groovy spript adds property a and b to finished build X and Y. (I use a and b for shorteness). Neither X not Y can see added a or b properties.
Only build Z could see added properties to X or Y

right?

If so, those parameters are unly used as like dep.btX.a in build Z.
This let you to add those parameters only for starting build Z.

In your groovy code that change would be in checking to only build number of a given SBuild, but in checking all properties link  dep.bt*.build.number and than reporting properties dep.bt*.a and dep.bt*.b

Will it work for you?

Hi Eugene, I don't understand completely what you are describing, but unfortunately I cannot afford to spend anymore time on the issue, and I've planned to port all the builds so that they publish the additional properties using the new feature you have introduced which relies on service messages. That's unfortunate, though, and I think it should have been pointed out in the upgrade notes.

Ok.

BTW, I implemented a patch to your code:

  public void addProductVersionAndAssemblyVersionParameters(BuildParameters parameters, SBuild build) {
    /*if(!build.isFinished())
    return;*/

    //name of property containing build number
    final String BUILD_NUMBER_PROP = "system.build.number";
    //dependency parameters prefix
    final String DEP_PREFIX = "dep.";

    for (Map.Entry<String, String> e : build.getParametersProvider().getAll().entrySet()) {
      String key = e.getKey();
      if (!key.startsWith(DEP_PREFIX)) continue;
      if (!key.endsWith(BUILD_NUMBER_PROP)) continue;

      //Here we find a property like dep.btA.system.build.number
      //Let's cut the prefix === dep.btA.
      final String prefix = key.substring(0, key.length() - BUILD_NUMBER_PROP.length());


      //call your code to compute properties
      String buildNumber = e.getValue();

      Pattern p = Pattern.compile(SOME_REGEX);

      Matcher m = p.matcher(buildNumber);

      if (!m.matches())
        return;

      if (m.groupCount() != 2)
        return;

      //add properties with prefix to build that depends on A
      parameters.addConfiguration(prefix + "build.number.productVersion", m.group(1));
      parameters.addConfiguration(prefix + "build.number.assemblyVersion", m.group(2));


      LOG.debug("Calculated productVersion and assemblyVersion configuration properties for build type " + build.getBuildTypeId());
    }
  }
 

Thank you very much Eugene.

I'm finally trying out service messages, but no luck, properties I set with service messages are not carried around onto dependent builds.
I see them in the build parameters tab of the build which generates them with the syntax ##teamcity[setParameter name='build.number.assemblyVersion' value='whatever'] but dependent builds which reference them won't start.
I have enabled back agent parameter publishing properties in the agent configuration files, do I have to turn them off for this to work?

Simone,
please check you have switched off the workaround I gave you to make your groovyplug work.
Do you see reported properties in build parameters tab?

Hi Eugene,

I removed the properties

teamcity.agent.publish.build.finish.properties=false
teamcity.agent.publish.build.start.properties=false

from the build agents configuration files, but it's not working.

In the build which generates the properties via service messages I correctly see what you can see in the attached picture, but a build which depends on that build and uses at least one of the properties will never start due to missing dependency property in the form dep.XXX.

Please try to add those confuration parameters to build configuration with value 'TBD'.

I've added an issue for that at
http://youtrack.jetbrains.net/issue/TW-17536
please vote for it.

Eugene, it works, that's why I pointed out that if I dump the properties and remove the dependency upon the property the build starts and the dependency properties are correctly dumped. What is missing is the issue you have opened, you cannot depend on properties generated during the build.
We cannot upgrade with this issue in place unfortunately.

You cannot depend on properties generated in the build unless you define that property name in the build configurations. This would work as TeamCity allows you to update propetry values in the build. The only reason to explicitly define property in the build configuration is to make parameters resolution on TeamCity know the build do contains all parameres.

Let's consider a small exaple:
I have build A that defines assemblyVersion property wia service message in the build.
I have build B that refers to dep.btA.asseblyVersion propertie

TeamCity would not allow running the build of B as it never know that dep.btA.assemblyVersion defined. But if you define assemblyVersion property in configration A it will work. You may notice on build parameters tab of A that assemblyVersion property value would be updated in the build to exactly the same value as you reported from the build as service message.

Does it work for you?

Hi Eugene, I understand completely what you say, but I don't agree that build B can't start unless I define the property on build configuration A explicitly. Say build B depends on build A by means of an artifact dependency which downloads artifacts from the last successful build of A. Build B just need to ensure that the latest successful build of A has that property defined, which is indeed the case here.
Leaving aside logic exercises that don't lead anywhere, this is a perfectly valid use case that used to work in 5.1 by means of the Groovy addin, and now is simply broken.
If it didn't come through clearly, we are stuck with 5.1 unless a fix for this is released, and we cannot just go through all the build configurations we have and define those properties explicitly; honestly, this issue has already wasted way more of my time than I'd hoped.

Hello,
Simone,
Sorry, that issue took to much time.

I want to summarize all workarounds that were suggersted in this discussion.  

To make sure applied workaournd would work, please restart TeamCity server to make TeamCity update
compatibility caches between build configurations.

     teamcity.agent.publish.build.finish.properties=true
     teamcity.agent.publish.build.start.properties=true
to buildAgent.properties files of every build agent.
This would make groovy plug script be called for every build instance (as it was in 6.0)

This workaround would only work if you re-run all dependent builds. TeamCity will call groovyplug extensions only
for build instances that were started on build agent with those properties set. For such builds you may notice
build parameters tab is almost empty and does not contain, say, teamcity.agent.name configuration parameter.

Please check your groovy script shoud add some property values in emulation mode
(see second argument of getParameters  method of GroovyPropertiesProvider). This is equivalent to
your build configuration has build number pattern that matches to your regex in groovyplug code.

Workaround 2. Make build configuraions report properties with service messages.
Use ##teamcity[setParameter name='assemblyVersion' value='some value']
To make builds compatible you need to add those parameters stubs to build configuration settings.
You will see those parameters in 'Build Parameters' tab written in bold with tooltip 'value changed'.
This is the most recommended way for migration, but, it involves too much changes.

Workaround 3. Make groovyplug script add dep.btA.* properties for build. For this case I've attached you
a code sample of a patch for your groovyplug code. I checked this code locally and it worked for me.
To make this code work you need to restart server or add/remove a build parameter for a configuration
that is not compatible with build a agent.
Note, same as in workaround 1, your groovyplug script should return those properties in emulation mode.

Workaround 4. Only for 6.5.2+. I added internal property into TeamCity to make it work in the same way as it
worked in 6.0. It do publishes all properies from build parameters, but it still calls groovyplug for
finished builds. To enable it you will need to upgrade to 6.5.2 and to set
teamcity.parametersProviders.callForFinishedBuilds=true
interenal property for server. See more infor on TeamCity internal properties at
http://confluence.jetbrains.net/display/TCD65/Configuring+TeamCity+Server+Startup+Properties


I hope, this summarized our discussion.
If you did a workaround and it failed to work, please let me know the detail.
We plan to release 6.5.2 in about a week.

Thanks Eugene, I appreciate it. Give me some some to go over your comments and I'll get back to you then.

Hi Eugene, thanks for the summary. Actually, the preferred solution for us in the long run would be to be able to use service messages without needing to declare stub properties in the build conf, what do you think?

Simone,

This seems to be right decision in the long run. That should make things as easy as possible.
Even more, this would not allow running a build if (for some reason) your propertries were not reported from a dependent build(s).

In the thread above, I created an issue for you
http://youtrack.jetbrains.net/issue/TW-17536
Seems this is what you are talking about.

I checked it once more with TeamCity 6.5 and it does requred to add parameters to build configuration on order to make it work (now).

On the other hand,
- You suggest to make TeamCity detect implicitly declared properties (= that were (but may not be) sent from build to service message)
- Now TeamCity works in a declarative way: as you like to use variable=property you have to declare it in a build configuration (same as in c#, java, etc)

Both approaches have it's prons and cons.

That is not clear, how should TeamCity work if there are 2 dependent builds:
- Older one has the reported (from script) propery
- Newer one does not have the reported (from script) property.
Let's suppose both are succeeded.

How do you think, which one shoud system take?

Hi Eugene, inline (although this boards makes it so difficult to quote a previous message and respond inline!)

eugene.petrenko wrote:

Simone,

This seems to be right decision in the long run. That should make things as easy as possible.
Even more, this would not allow running a build if (for some reason) your propertries were not reported from a dependent build(s).

Exactly, that's how it is working now by using the Groovy plugin.

In the thread above, I created an issue for you

http://youtrack.jetbrains.net/issue/TW-17536

Seems this is what you are talking about.

Yes, that is it

I checked it once more with TeamCity 6.5 and it does requred to add parameters to build configuration on order to make it work (now).

Exactly, that's the issue we're having

On the other hand,

- You suggest to make TeamCity detect implicitly declared properties (= that were (but may not be) sent from build to service message)

- Now TeamCity works in a declarative way: as you like to use variable=property you have to declare it in a build configuration (same as in c#, java, etc)

Well, the Groovy addin already lets you achieve this scenario, I suggest to implement the same behavior for properties declared via service messages. I am probably missing some of the internals about how TC works, but this is the desired behavior. In particular, I do not agree with the behavior you described somewhere else that properties are declared on a build configuration (assuming I understood correctly), instead I think they should belong to a specific build. Each build should have its own properties, eventually inherited from the build configuration. I think this modeling would comply very well with the scenario we are talking about.

Both approaches have it's prons and cons.

That is not clear, how should TeamCity work if there are 2 dependent builds:

- Older one has the reported (from script) propery

- Newer one does not have the reported (from script) property.

Let's suppose both are succeeded.

How do you think, which one shoud system take?

If older one has the required properties and the newer on does not I think the dependent build should not start in the general case, but I guess it would depend on the permutations of the dependencies (snapshot, artifact) you have configured. Nonetheless I think _not starting_ is a sensible default.

Simone,

I just realized, you may make groovy plugin to report same properties as you report via service messages for enumation mode.
It would be a good workaroung to avoid adding those properties to build configuration.

This would make TeamCity believe those properties are defined.
If you decide to try it, please restart teamcity to make compatibility cache flushed.

Hi Eugene, I'd prefer to use either one or the other way, not both at the same time. I noticed the update to the issue report, unfortunately we're using artifact and snapshot dependencies together. We'll postpone the upgrade until this is fixed.