NuGet Dependency Trigger - Quiet Period?

Hi,

Is there a way to configure a quiet period to control the frequency of checks against the feed in the NuGet Dependency Trigger (similar to the VCS trigger)?  Our current build system has several hundred triggers across 110 build configurations and is choking a bit under the load, considering the dependency trigger runs solely on the TeamCity server (and not the agents).

Furthermore, following some minor investigation it seems that a new NuGet.exe process is spawned for each NuGet dependency trigger instance (please correct me if I'm wrong?).  It would seem that this design doesn't scale well to an enterprise configuration with hundreds of triggers.  The preference here would be to batch all the triggers within a build configuration to a single NuGet.exe list process and check the Package ID's then - I guess this would be the ideal solution instead of exposing a quiet period...

Hope that makes sense... and thanks in advance,

Steve

17 comments
Comment actions Permalink

Thank you for feedback. There were an initial implementation of NuGet triggering. I plan to improve it in the next builds. I'll let you know when build will be ready.

Could you please describe the number of build triggers you have per configuration/total?
What NuGet feed to you use? Do you have many triggers targeted to the same NuGet Feed?

I added an issue for that problem at http://youtrack.jetbrains.net/issue/TW-18429

0
Comment actions Permalink

Hi Eugene,

Some numbers regarding the build triggers.  These are from memory, but they should be pretty accurate!

Per configuration: between 1 and 5 triggers
Total: Around 120 triggers (this will scale upwards, with an expected maximum around 500-600 when all builds implemented)
NuGet Feed: Only internal feeds, currently 2, triggers split evenly between both feeds.  Feeds currently based on an Orchard NuGet.Gallery implementation.

Regards,

Ben

0
Comment actions Permalink

How may different packages you have?

0
Comment actions Permalink

Approximately 190 different package ids per feed, two feeds, but this will grow and may end up at somewhere in the thousands.  Each package id can have anything from 1-5 versions (unusual) to upwards of 50 (more common).

0
Comment actions Permalink

Hi Eugene,

Thanks for your quick response to this.  I'm about to start testing the latest build of the plug-in to see if all is OK - I'll let you know if any issues.

Steve

0
Comment actions Permalink

Hi Eugene,

Is this the best place to post feedback from testing?  I'll assume so for now...

Am having issues with the latest build.  I'm able to run a list command via command prompt when logged in as the TeamCity service account user successfully, but getting errors when the dependency trigger runs.  Any help you can provide would be well received.

Here's the stack trace from teamcity-server.log.  Please note we're testing against a private nuget feed built from https://github.com/NuGet/NuGetGallery.git, which uses a different data service endpoint url (you'll notice our endpoint is http://beta.nuget.dev.cba/DataServices/Feeds.svc/).

Thanks in advance.  Steve


[2011-10-07 12:03:52,276]   WARN - r.impl.PackageCheckerNuGetBulk - Failed to bulk check changes of NuGet packages. Failed to execute NuGet TeamCity.ListPackages command. Exited code was 1 jetbrains.buildServer.nuget.server.exec.NuGetExecutionException: Failed to execute NuGet TeamCity.ListPackages command. Exited code was 1      at jetbrains.buildServer.nuget.server.exec.NuGetOutputProcessorAdapter.onFinished(NuGetOutputProcessorAdapter.java:53)      at jetbrains.buildServer.nuget.server.exec.impl.NuGetExecutorImpl.executeNuGet(NuGetExecutorImpl.java:74)      at jetbrains.buildServer.nuget.server.exec.impl.ListPackagesCommandImpl.checkForChanges(ListPackagesCommandImpl.java:86)      at jetbrains.buildServer.nuget.server.trigger.impl.PackageCheckerNuGetBulk$1.run(PackageCheckerNuGetBulk.java:74)      at jetbrains.buildServer.util.ExceptionUtil$1.run(ExceptionUtil.java:37)      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)      at java.util.concurrent.FutureTask.run(Unknown Source)      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)      at java.lang.Thread.run(Unknown Source) [2011-10-07 12:08:53,855]   WARN - impl.ListPackagesCommandImpl$1 - Failed to execute commnad: Could not connect to the feed specified at 'http://beta.nuget.dev.cba/DataServices/Feeds.svc/'. Please verify that the package source (located in the Package Manager Settings) is valid and ensure your network connectivity. System.InvalidOperationException: Could not connect to the feed specified at 'http://beta.nuget.dev.cba/DataServices/Feeds.svc/'. Please verify that the package source (located in the Package Manager Settings) is valid and ensure your network connectivity. ---> System.Data.Services.Client.DataServiceQueryException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException: <?xml version="1.0" encoding="utf-8" standalone="yes"?> <error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">   <code></code>   <message xml:lang="en-US">Recursion reached allowed limit.</message> </error>    at System.Data.Services.Client.QueryResult.Execute()    at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)    --- End of inner exception stack trace ---    at System.Data.Services.Client.DataServiceRequest.Execute[TElement](DataServiceContext context, QueryComponents queryComponents)    at System.Data.Services.Client.DataServiceQuery`1.Execute()    at System.Data.Services.Client.DataServiceQuery`1.ExecuteInternal()    at System.Data.Services.Client.DataServiceQuery.Execute()    at NuGet.DataServiceQueryWrapper`1.Execute[TResult](Func`1 action)    --- End of inner exception stack trace ---    at NuGet.DataServiceQueryWrapper`1.Execute[TResult](Func`1 action)    at NuGet.DataServiceQueryWrapper`1.<GetAll>d__7.MoveNext()    at JetBrains.TeamCity.NuGet.ExtendedCommands.NuGetTeamCityListPackagesCommand.ExecuteCommandImpl() in d:\teamcity\work\ff38610775f96270\nuget-extensions\nuget-commands\src\NuGetTeamCityListPackagesCommand.cs:line 41    at JetBrains.TeamCity.NuGet.ExtendedCommands.CommandBase.ExecuteCommand() in d:\teamcity\work\ff38610775f96270\nuget-extensions\nuget-commands\src\CommandBase.cs:line 13 TeamCity command failed

0
Comment actions Permalink

Steve,

Thank you for feedback. I belive the best place to report issues is our issue tracker: http://youtrack.jetbrains.net. Feel free to create issue for TeamCity project.


I checked the issue. It seems I joined too many packages in one call to NuGet server. Could you please try to capture HTTP request from TeamCity nuget to nuget feed and check how long was the query? I generate basic quest like: Where(pkg => (pkg.Id  ==  'aaa') || ... || (pkd.Id == 'zzz')). The list of ors could be too long.

I'll add a setting to limit the number of packages that were jouned for one feed.

UPD: From the other side, this issue could also be interesting to NuGet feed project contributors. If you posted the question to them, please let me know the link.

0
Comment actions Permalink

Steven,

Please try the next build. It will now limit number of packages to check to 30 por one call.

Message was edited by: Eugene Petrenko

0
Comment actions Permalink

Eugene,

Thanks for the update, I'll be running some tests today (based in Sydney hence the time diff).  I'm unsure of the differences in the odata feeds in the new NuGetGallery and the original Orchard Gallery, but it certainly looks like a query length issue based on the "Recursion reached allowed limit." message.

Will keep you posted - will also start a discussion on the NuGetGallery github side...

Thanks again
Steve

0
Comment actions Permalink

Steve, have you tried the settings? Does it work for you now?

0
Comment actions Permalink

Hi Eugene,

We have been testing the triggers for a while now, there are a few weird issues where occasionally we seem to get triggers firing when the underlying package feed has not updated for the specific Id we are triggering on.  We are just keeping an eye on it at the moment to see if we can get a better idea of exactly what circumstances lead to an incorrect trigger event.

Other than that, it is looking very good.  Doesnt appear to have any of the load issues that we previously encountered.

Thanks,

Ben

0
Comment actions Permalink

Ben,

Thanks for good news.

The false triggerings could happen because of network connection errors. As there was a error, NuGet plugin removes package state and thus as connection is up again, it triggers the build. Please check server logs for errors from the trigger to ensure that was the issue.

0
Comment actions Permalink

Ah, interesting.  That may explain what we have been seeing.  So, just to be sure I understand:

1) If the extension gets an error on trigger check, state for that trigger is deleted.
2) On next trigger, it is assumed that if there are no errors that the trigger should fire a build off.

If that is (somewhat) accurate, what would happen when the TeamCity server is restarted (ie no error state on a trigger)?  Does the state of the package triggers get persisted (ie version, id, source), or does this then cause the trigger to fire?

I am wondering, if we can assume that a package is generally uniquely identifid by the version, Id and source, whether this can be used to persist the "state" of the package across both restarts and errors?  We had a few network glitches today, and we saw quite large build storms occuring due to low level packages being triggered when no change on the package feed had actually occurred.  Of course, as soon as we have a build that triggers and publishes due to a network outage, we get a lot that will trigger due to valid publish events!

Will have another look through the logs to confirm tomorrow.

0
Comment actions Permalink

I checked the code once more. There should be no build starts after connection error or server restart.

Java side of the plugin does not have any logic to compare packages version, thus, it uses a list of all matching packages (sorted in some fixed order) as a version hash.
This means, it may trigger a build if you remove some (manching, but not necessarily latest) package from the feed.
Does it looks like your case?

0
Comment actions Permalink

We are still having difficulty establishing the exact behaviour that we are seeing.  It appears that we are getting triggers firing on packages that are definitely not changing, no additions or removals.

Can you point me to the code the location in the Java code that is doing the comparison?  Might take a look and see if there is anything obvious...

0
Comment actions Permalink

Finally I found the issue in comparison. Please try new build (#0.5.1052.5 or newer)

0

Please sign in to leave a comment.