Issue with TeamCity Nuget feed after re-indexing

Permanently deleted user

Created February 15, 2015 03:14

Hello.

I've upgraded my server to 9.0.2 and that resulted in a re-indexing of the builds, Afterwards, the TeamCity Nuget feed became troublesome. Although browsing and selecting the desired version works ok, one of our deployment tools (Red Gate Deployment Manager) fails to resolve which version to choose. When trialing the update process, we skipped the re-indexing and the NuGet feed worked as before.

Here is the error I am seeing :

[02:19:20]Getting package versions for each step...

[02:19:20]There are no packages available for step 6eda33a2-aa80-47de-99b9-a23981172dd4 (Skyscanner.IisDeploymentFramework.Com (Deploy Skyscanner.IisDeploymentFramework.Com))

[02:19:20]System.InvalidOperationException: There are no packages available for step 6eda33a2-aa80-47de-99b9-a23981172dd4 (Skyscanner.IisDeploymentFramework.Com (Deploy Skyscanner.IisDeploymentFramework.Com))

[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Extensions.ProjectExtensions.GetLatestPackageForStep(IDeploymentManagerSession session, PackageStepResource step)

[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.CreateReleaseCommand.GetSelectedPackages(IList`1 steps)

[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.CreateReleaseCommand.GetRelease(ProjectResource project)

[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.DeployCommandBase.ExecuteImpl()

[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.ApiCommand.Execute()

[02:19:20] at RedGate.Deploy.Shared.Startup.CommandProcessor.<ProcessAsync>d__4.MoveNext()

The odd thing is that if I create the release manually (ie, choose which package versions to use instead of just grabbing the latest), I can request deployments and the deployment tool has no issue retrieving the artifact.

The issue is definitely on TeamCity side because if I route the artifacts through our Artifactory instance, there is no problem. However, this solution is not feasible as the projects that still use the Red Gate Deployment Manager are databases and .NET applications under intensive care for a few months.

Any ideas on how to tackle this problem?

4 comments

Alina Mishina

Created February 17, 2015 23:07

Hi Paul,

How do you grad the latest package using Red Gate?
Please try to change teamcity.nuget.api.version to v1 (see the related comment).
If it does not help then please attach teamcity-server.log with debug enabled and nuget feed request processing captured.

Permanently deleted user

Created February 18, 2015 12:35

Hello, Alina.

Unfortunately, this is a licensed, closed-source product and we do not know how it works internally. We have solved the issue by going through the API instead of using their command-line tool (will post the Powershell code if anybody is interested). Thank you for the suggestion of using v2, I was not aware it was available.

We also hit build patch issues after the upgrade the first work day after the upgrade, ie

[11:25:25]Failed to build patch for build #1.0.64219.1122 {build id=3485033, buildTypeId=CommonUserExperience_WebApplications_WebsiteScaffoldingOctopus_11BuildAndPackage}, VCS root: "Website Scaffolding" {instance id=2233, parent internal id=692, parent id=WebsiteScaffoldingOctopus, description: "svn: http://svn.corp.skyscanner.local/svn/SkyscannerWebsite/SEO"}, due to error: Cannot build patch: java.lang.NullPointerException

[...]

[11:25:47]Patch is broken, can be found in file: G:\TeamCity\BuildAgent\temp\globalTmp\temp1067199266517313017patch_3485033

[11:25:47]Failed to build patch for build #1.0.64219.1122 {build id=3485033, buildTypeId=CommonUserExperience_WebApplications_WebsiteScaffoldingOctopus_11BuildAndPackage}, VCS root: "Website Scaffolding" {instance id=2233, parent internal id=692, parent id=WebsiteScaffoldingOctopus, description: "svn: http://svn.corp.skyscanner.local/svn/SkyscannerWebsite/SEO"}, due to error: Cannot build patch: java.lang.NullPointerException

[11:25:47]
jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl$1: Server was not able to build correct patch, most likely due to VCS errors at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.throwError(PatchDownloaderImpl.java:118) at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatch(PatchDownloaderImpl.java:108) at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.copyPatchAndCheck(PatchDownloaderImpl.java:69) at jetbrains.buildServer.agent.impl.patch.UpdateSourcesPatcherBase.copyPatchToTempFile(UpdateSourcesPatcherBase.java:71) at jetbrains.buildServer.agent.impl.patch.UpdateSourcesFromServer.updateSources(UpdateSourcesFromServer.java:60) at jetbrains.buildServer.agent.impl.patch.UpdateSourcesBuildStageBase.doSourceUpdate(UpdateSourcesBuildStageBase.java:91) at jetbrains.buildServer.agent.impl.patch.UpdateSourcesBuildStageBase.doRecoverableStage(UpdateSourcesBuildStageBase.java:59) at jetbrains.buildServer.agent.impl.buildStages.startStages.RecoverableBuildStage.doLastAttempt(RecoverableBuildStage.java:112) at jetbrains.buildServer.agent.impl.buildStages.startStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:70) at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31) at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24) at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78) at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37) at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24) at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:70) at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:50) at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:266) at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:51) at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:231) at java.lang.Thread.run(Thread.java:744) Caused by: jetbrains.buildServer.vcs.patches.UnsuccessfulPatchException: Failed to build patch for build #1.0.64219.1122 {build id=3485033, buildTypeId=CommonUserExperience_WebApplications_WebsiteScaffoldingOctopus_11BuildAndPackage}, VCS root: "Website Scaffolding" {instance id=2233, parent internal id=692, parent id=WebsiteScaffoldingOctopus, description: "svn: http://svn.corp.skyscanner.local/svn/SkyscannerWebsite/SEO"}, due to error: Cannot build patch: java.lang.NullPointerException at jetbrains.buildServer.vcs.patches.AbstractPatcher$1.fail(AbstractPatcher.java:93) at jetbrains.buildServer.vcs.patches.LowLevelPatcher.readPatchStream(LowLevelPatcher.java:165) at jetbrains.buildServer.vcs.patches.LowLevelPatcher.applyPatch(LowLevelPatcher.java:88) at jetbrains.buildServer.vcs.patches.AbstractPatcher.applyPatch(AbstractPatcher.java:42) at jetbrains.buildServer.agent.impl.patch.PatchApplierImpl.applyPatch(PatchApplierImpl.java:18) at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatchInFileIsCompleted(PatchDownloaderImpl.java:88) at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatch(PatchDownloaderImpl.java:98)

To fix this issue, we have tried :

Clean checkout directory (from the custom build interface) - no effect
Clean checkout (using the Actions -> Enforce Clean Checkout) - temporarily fixed the issues, usually for a few builds, then consistently broken again on all agents
Clean caches : vcsContentCache, svn, git, then agentFiles (from the Administration -> Diagnostics interface) - no effect
Stop server service, delete contents of %TC_DATA_DIR%\system\caches, start service - no effect
Systemally delete all workspaces for all Windows agents (Linux agents were not affected by this issue) - limited effect, still seeing this issue on over half of the build configurations
Updated the jre to 7u75 (x64) - no effect

Before upgrading, we have succesfully tested the upgrade process on a full clone of the production server, skipping only the indexing to save time.The fact that a simple reindexing has caused so much trouble and there wasn't even a hint about it in the upgrade documentation is very concerning and the discussion I started about making the TeamCity upgrade a regular occurence has now shifted towards decentralisation and if TeamCity is becoming a single point of failure for our CD model.

Alina Mishina

Created February 19, 2015 17:40

Hi Paul,

Most likely this is the following issue, caused by incorrect/problematic upgrade: https://youtrack.jetbrains.com/issue/TW-39810
(copied from email thread)

Permanently deleted user

Created February 19, 2015 19:21

Thank you very much for pointing me to that bug report. Deleting svnkit-1.8.4.jar, svnkit-cli-1.8.4.jar and svnkit-javahl16-1.8.4.jar solved the problem.

Please sign in to leave a comment.