Problem with Maven Artifact Dependency Trigger

Hello,

I'm trying to set up an artifact trigger for Maven project. The artifact is stored in our local Maven repository. Repository is hidden behind https and requires authorization.

At first I was having issues with accessing it during the build process for dependency resolving. I added "-Djavax.net.ssl.trustStore=<path_to_jks>" to the JVM command line parameters of the build configuration.This helped and TeamCity was able to download all the dependencies and complete the build without any issues.

Then, I decided to add "Maven Artifact Dependency Trigger" for one of the dependencies and now I can see the following message "Unable to check for Maven dependency updates: Not authorized." In teamcity-vcs.log I can see the following:

[2011-11-10 15:04:52,100]   WARN [prio executor 9] - MavenArtifactDependencyTrigger - unable to check for updates in Maven artifact <groupId>:<artifactId>:<packaging>:<version>[<Maven_Repository_URL>] for build configuration Project :: Build_name {id=bt4}: Not authorized.

Some additional info that may be necessary. $M2_HOME/conf/ directory does not contain settings.xml file. All the settings are set in ~/.m2/settings.xml of user that is running the server. I encounter this problem both on our production server which is running TeamCity Enterprise 6.0.1 (build 15816) and my test server which is running TeamCity Professional 6.5 (build 17795).

Can you please assist me?

23 comments
Comment actions Permalink

Hello Andrew,

I apologize for the delay.

Can I ask you to add following line to internal.properties (see here http://confluence.jetbrains.net/display/TCD7/Configuring+TeamCity+Server+Startup+Properties):

teamcity.maven.settingsPath=<put the full path to maven settings.xml here>

and restart the server.

After this, please try Maven artifact trigger once again.
This is to make sure TeamCity uses correct settings.xml file while connecting to repository.

0
Comment actions Permalink

Hello Nikita,

Thanks for your reply. I did as you requested:

teamcity@UbuntuVM:/opt/TeamCity-6.5.5/logs$ cat ~/.BuildServer/config/internal.properties
teamcity.maven.settingsPath=/home/teamcity/.m2/settings.xml


And as I already mentioned, there is no settings.xml file in the Maven home directory:

teamcity@UbuntuVM:/$ ls -l /opt/maven/conf/settings.xml
ls: cannot access /opt/maven/conf/settings.xml: No such file or directory


I also had to set the following property in order for TC to be able to download artifacts from our repository:

teamcity@UbuntuVM:/opt/TeamCity-6.5.5/logs$ echo $TEAMCITY_SERVER_OPTS
-Djavax.net.ssl.trustStore=/opt/maven/conf/maven.jks


But the error is still there... I turned on 'debug-general' in server settings and in teamcity-server.log I can see the following:

[2011-11-22 12:21:34,132]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Found 0 components to load on start
[2011-11-22 12:21:34,133]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Building Maven global-level settings from: '/opt/maven/conf/settings.xml'
[2011-11-22 12:21:34,133]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Building Maven user-level settings from: '/home/teamcity/.m2/settings.xml'
[2011-11-22 12:21:34,134]  DEBUG - avenArtifactChangeDetectorImpl - Repository lookup order: [Repository[_tc_temp_remote_repo|https://<our_repository_url>]]
[2011-11-22 12:21:34,135]  DEBUG - avenArtifactChangeDetectorImpl - Detecting changes in artifact <groupId>:<artifactId>:<packaging>:<version>:<scope>
[2011-11-22 12:21:34,136]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven INFO] snapshot <groupId>:<artifactId>:<packaging>:<version>: checking for updates from _tc_temp_remote_repo
[2011-11-22 12:21:34,136]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Checking for pre-existing User-Agent configuration.
[2011-11-22 12:21:34,136]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Adding User-Agent configuration.
[2011-11-22 12:21:34,136]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Connecting to repository: '_tc_temp_remote_repo' with url: '<our_repository_url>'.
[2011-11-22 12:21:34,204]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven WARN] repository metadata for: 'snapshot <groupId>:<artifactId>:<packaging>:<version>' could not be retrieved from repository: _tc_temp_remote_repo due to an error: Authorization failed: Not authorized.
[2011-11-22 12:21:34,204]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Exception
org.apache.maven.wagon.TransferFailedException: Authorization failed: Not authorized.
...


At the same time when I launch the build, TC successfully checks for artifacts' snapshot dependencies (which are located in the local repository) and downloads them (if necessary) without any issues. And that is really confusing...

0
Comment actions Permalink

Hello Andrew,

Thank you for the logs.

Some questions to check, so I can further try to reproduce the issue on our side:

Are you using any type of client SSL authentication with your local repository?
What type of authentication (if any) is used by repository server? where are credentials stored?

0
Comment actions Permalink

Sorry for delay in my reply. We use LDAP authentication and all the credentials are stored on our LDAP server.

Does this help? If no, can you please specify some additional information so I could clarify this question with our sysadmins team.

0
Comment actions Permalink

Andrew,

In the first post you wrote "Repository is hidden behind https and requires authorization.". I'm interested in "authorization" part.

What kind of authorization do you use in your build to access maven repository? A username/password pair? Client-side certificates?

0
Comment actions Permalink

Credentials (username and password) are stored in $HOME/.m2/settings.xml file, <servers> section.
Usually, password is encrypted (http://maven.apache.org/guides/mini/guide-encryption.html), but I've also tried with plain password and result is the same.

0
Comment actions Permalink

Hello Andrew,

I apologize for the delay: I have not yet reproduced the issue - have some problems setting up the environment.

Meanwhile, could you please try using TeamCity 6.5.5 ? It is known to contain fixes relevant to environmental variables availability to Maven instance on TC server.

0
Comment actions Permalink

Hello Nikita,

No worries, I can wait =)

I've already tried 6.5.5. No luck, unfortunately.

0
Comment actions Permalink

Hi Andrew,

The problem doesn't appear to relate to https. We were able to reproduce this with the simple basic authorization. I'm taking over this issue.

0
Comment actions Permalink

Sergey, good to hear that! I'll be awaiting for the news then.

0
Comment actions Permalink

Hi Andrew,

Good news.

The problem is that TeamCity always creates a new repository record with id "_tc_temp_remote_repo" if repository URL is specified in the trigger settings. Obviously you don't have credentials in your settings.xml for this id. That's why requests fail.

To overcome that you have the following options :

1) add credentials for "_tc_temp_remote_repo" in your settings.xml. Not very good options because "_tc_temp_remote_repo" is shared between all triggers. Also this is an internal name that may change in the future.

2) If the trigger is attached to a Maven build configuration, you can leave repository URL empty. Then TeamCity will take the repository list from the project model with all their credentials. If TeamCity is able to read the model, of course. It's easy to check on the Maven 2 tab.

From your previous posts I can conclude you have set up a Maven configuration. So I suggest you the second option.

Please let me know if this helps.

0
Comment actions Permalink

Hello Sergey,

Removing the URL fixed the authorization problem, thank you!

However, the build doesn't trigger after the artifact is updated in repository. Here is the log snippet:

[2011-12-07 17:52:31,563]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] <artifactId>: resolved to version TRUNK-20111207.133621-164 from repository <repositoryId>
[2011-12-07 17:52:31,563]  DEBUG - maven.acd.WagonDigestRetriever - Retrieving digest for <groupId>:<artifactId>:pom:TRUNK-SNAPSHOT:compile
[2011-12-07 17:52:31,563]  DEBUG - maven.acd.WagonDigestRetriever - Looking into Repository[<repositoryId>|https://<repositoryUrl>]
[2011-12-07 17:52:31,564]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Checking for pre-existing User-Agent configuration.
[2011-12-07 17:52:31,564]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Adding User-Agent configuration.
[2011-12-07 17:52:31,564]  DEBUG - .maven.acd.DefaultWagonSession - downloading <path>/<artifactId>-TRUNK-20111207.133621-164.pom.sha1 into /mnt/TC/TeamCity-7.0/temp/_tc_maven7499358735004759566checksum
[2011-12-07 17:52:31,798]  DEBUG - .maven.acd.DefaultWagonSession - done
[2011-12-07 17:52:31,798]  DEBUG - maven.acd.WagonDigestRetriever - SHA-1 digest found
[2011-12-07 17:52:31,798]  DEBUG - maven.acd.WagonDigestRetriever - Found: 5d459520fad1061760584551c786acaa6aed09e7 SHA-1
[2011-12-07 17:52:31,798]  DEBUG - .buildServer.maven.acd.Checker - Retrieved digest: 5d459520fad1061760584551c786acaa6aed09e7 SHA-1
[2011-12-07 17:52:31,798]  DEBUG - .buildServer.maven.acd.Checker - Currently stored digest: 5d459520fad1061760584551c786acaa6aed09e7
[2011-12-07 17:52:31,799]  DEBUG - .buildServer.maven.acd.Checker - Result: false

And this is when the artifact gets updated:

[2011-12-07 17:54:38,532]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] <artifactId>: resolved to version TRUNK-20111207.143130-165 from repository <repositoryId>
[2011-12-07 17:54:38,532]  DEBUG - maven.acd.WagonDigestRetriever - Retrieving digest for <groupId>:<artifactId>:pom:TRUNK-SNAPSHOT:compile
[2011-12-07 17:54:38,532]  DEBUG - maven.acd.WagonDigestRetriever - Looking into Repository[<repositoryId>|https://<repositoryUrl>]
[2011-12-07 17:54:38,533]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Checking for pre-existing User-Agent configuration.
[2011-12-07 17:54:38,533]  DEBUG - ven.DefaultMavenEmbedderLogger - [Maven DEBUG] Adding User-Agent configuration.
[2011-12-07 17:54:38,534]  DEBUG - .maven.acd.DefaultWagonSession - downloading <path>/<artifactId>-TRUNK-20111207.143130-165.pom.sha1 into /mnt/TC/TeamCity-7.0/temp/_tc_maven8958822966802763004checksum
[2011-12-07 17:54:38,878]  DEBUG - .maven.acd.DefaultWagonSession - done
[2011-12-07 17:54:38,878]  DEBUG - maven.acd.WagonDigestRetriever - SHA-1 digest found
[2011-12-07 17:54:38,878]  DEBUG - maven.acd.WagonDigestRetriever - Found: 5d459520fad1061760584551c786acaa6aed09e7 SHA-1
[2011-12-07 17:54:38,878]  DEBUG - .buildServer.maven.acd.Checker - Retrieved digest: 5d459520fad1061760584551c786acaa6aed09e7 SHA-1
[2011-12-07 17:54:38,878]  DEBUG - .buildServer.maven.acd.Checker - Currently stored digest: 5d459520fad1061760584551c786acaa6aed09e7
[2011-12-07 17:54:38,878]  DEBUG - .buildServer.maven.acd.Checker - Result: false

Maybe it's connected to the fact that this is a pom artifact? And if the contents haven't changed, hash shouldn't change either (if it's solely based on contents of the file).

Please note that I'm testing 7.0 EAP version. I haven't tried it on 6.5.5 yet.

0
Comment actions Permalink

You're right. Artifact trigger considers an artifact updated when its digest changes. In your case the digest hasn't been changed.

0
Comment actions Permalink

Ok, I see. I wanted to use this trigger instead of Finish Build Trigger, but it won't work this way. Do you have any plans to make triggers customizable via build parameters?

My use case is as follows. We have Core part of the system and several Components that depend on various modules of core part. For each of the pairs Component + Core I want to set up a template that would trigger the build when Core part has been updated (e.g. its snapshot has been deployed to local Maven repository). Since we have many versions of both Core and Components, it would be great to make custom configurations based on a template (that doesn't change) and build parameters (that do change), instead of manually specifying this trigger for each configuration.

0
Comment actions Permalink

Andrew, I think it is possible with the snapshot artifact trigger. When it's enabled it triggers a build each time any of its SNAPSHOT dependencies is updated. Did you try it?

0
Comment actions Permalink

I'm testing it at the moment. But the build doesn't trigger when the snapshot dependency is built.

I assume it may be using some sophisticated algorightm based on pom.xml analysis. Say, if project A has 3 modules A1, A2 and A3, project B depends on A2 module and if I make changes to A1 only and deploy all artifacts of project A to repository, the build won't trigger?

My question though is if you plan to add some customization to these triggers. Because at the present moment I have to manually bind each of "Component + Core" build configurations triggers to the corresponding "Core snapshot" configuration (using finish build trigger).

0
Comment actions Permalink

> Say, if project A has 3 modules A1, A2 and A3, project B depends on A2 module and if I make changes to A1 only and deploy all artifacts of project A to repository, the build won't trigger?

No, it won't. Nothing really sofisticated. As I mentioned before an artifact is considered updated if its content changes. The trigger simply collects all external SNPASHOT dependencies and checks each of them.

> My question though is if you plan to add some customization to these triggers. Because at the present moment I have to manually bind each of "Component + Core" build configurations triggers to the corresponding "Core snapshot" configuration (using finish build trigger).

Actually, the artifact trigger can be customized (to some extent). In the template you can define a trigger that will use config parameters (variables).

For example:

[in the template]

define an artifact trigger for: %groupId%:%artifactId%:%artifactVersion%


[in the build configuration 1 based on the template]

define config parameters:
groupId=someGroup1
artifactId=someArtifact1
artifactVersion=version1

[in the build configuration 2 based on the template]

define config parameters:
groupId=someGroup2
artifactId=someArtifact2
artifactVersion=version2



and so on.


But I think the snapshot artifact trigger suits your goals better.
0
Comment actions Permalink

Forgot to mention one more option - snapshot dependencies (a TeamCity term, not Maven).

The idea is to define snapshot dependencies between components and the core. In the template you can configure the VCS trigger to "Trigger on changes in snapshot dependencies".

This will cause building all dependent components on changes in the core with proper build ordering.

0
Comment actions Permalink

Ok, I see. Thank you for the explanation. Looks like snapshot artifact trigger is going to be the choice.

One last question. In "Build Triggering" I can set "Trigger on changes in shapshot dependencies" in "VCS Trigger" (as you suggested in your last post) and I can also add "Maven Snapshot Dependencies Trigger". My guess is that the first one detects a commit in snapshot dependency and triggers its build and then the dependant build. And the second one waits for snapshot dependency to be updated in the Maven repository and then triggers the dependant build. Is this correct?

In any case, I won't be able to add snapshot dependency to template because for each "Core + Component" version pair its going to be different (based on "Core" version, which is produced by specific build configuration of "Core snapshot").

0
Comment actions Permalink

> One last question. In "Build Triggering" I can set "Trigger on changes in shapshot dependencies" in "VCS Trigger" (as you suggested in your last post) and I can also add "Maven Snapshot Dependencies Trigger". My guess is that the first one detects a commit in snapshot dependency and triggers its build and then the dependant build. And the second one waits for snapshot dependency to be updated in the Maven repository and then triggers the dependant build. Is this correct?

In fact VCS trigger detects a commit in a snapshot dependency and triggers (puts to the queue) all the downstream builds simultaneously. But a dependant build won't go to an agent unless all its dependency builds finish.

You don't need to have both VCS trigger with "Trigger on changes in shapshot dependencies" and Snapshot Artifact Trigger activated since they do essentially the same thing. However VCS trigger in some cases works faster since it queues all the builds at once, while Snapshot Artifact Trigger queues builds one by one only when the upstream build finishes. This also means that Snapshot Artifact Trigger may cause unnesessary builds when you have multiple upstream builds.

> In any case, I won't be able to add snapshot dependency to template because for each "Core + Component" version pair its going to be different (based on "Core" version, which is produced by specific build configuration of "Core snapshot").
OK, I see

0
Comment actions Permalink

Thank you for the clarification. I guess this topic can be closed.

0
Comment actions Permalink

Just for traceability. This is the ticket for this issue - http://youtrack.jetbrains.net/issue/TW-19307

0

Please sign in to leave a comment.