Issue with TeamCity Nuget feed after re-indexing

Hello.

I've upgraded my server to 9.0.2 and that resulted in a re-indexing of the builds, Afterwards, the TeamCity Nuget feed became troublesome. Although browsing and selecting the desired version works ok, one of our deployment tools (Red Gate Deployment Manager) fails to resolve which version to choose. When trialing the update process, we skipped the re-indexing and the NuGet feed worked as before.

Here is the error I am seeing :

[02:19:20]Getting package versions for each step...
[02:19:20]There  are no packages available for step 6eda33a2-aa80-47de-99b9-a23981172dd4  (Skyscanner.IisDeploymentFramework.Com (Deploy  Skyscanner.IisDeploymentFramework.Com))
[02:19:20]System.InvalidOperationException:  There are no packages available for step  6eda33a2-aa80-47de-99b9-a23981172dd4  (Skyscanner.IisDeploymentFramework.Com (Deploy  Skyscanner.IisDeploymentFramework.Com))
[02:19:20] at  RedGate.Deploy.DeploymentManagerTools.Extensions.ProjectExtensions.GetLatestPackageForStep(IDeploymentManagerSession  session, PackageStepResource step)
[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.CreateReleaseCommand.GetSelectedPackages(IList`1 steps)
[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.CreateReleaseCommand.GetRelease(ProjectResource project)
[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.DeployCommandBase.ExecuteImpl()
[02:19:20] at RedGate.Deploy.DeploymentManagerTools.Commands.ApiCommand.Execute()
[02:19:20] at RedGate.Deploy.Shared.Startup.CommandProcessor.<ProcessAsync>d__4.MoveNext()


The odd thing is that if I create the release manually (ie, choose which package versions to use instead of just grabbing the latest), I can request deployments and the deployment tool has no issue retrieving the artifact.

The issue is definitely on TeamCity side because if I route the artifacts through our Artifactory instance, there is no problem. However, this solution is not feasible as the projects that still use the Red Gate Deployment Manager are databases and .NET applications under intensive care for a few months.

Any ideas on how to tackle this problem?

4 comments
Comment actions Permalink

Hi Paul,

How do you grad the latest package using Red Gate?
Please try to change teamcity.nuget.api.version to v1 (see the related comment).
If it does not help then please attach teamcity-server.log with debug enabled and nuget feed request processing captured.

0
Comment actions Permalink

Hello, Alina.

Unfortunately, this is a licensed, closed-source product and we do not know how it works internally. We have solved the issue by going through the API instead of using their command-line tool (will post the Powershell code if anybody is interested). Thank you for the suggestion of using v2, I was not aware it was available.

We also hit build patch issues after the upgrade the first work day after the upgrade, ie

[11:25:25]Failed  to build patch for build #1.0.64219.1122 {build id=3485033,  buildTypeId=CommonUserExperience_WebApplications_WebsiteScaffoldingOctopus_11BuildAndPackage},  VCS root: "Website Scaffolding" {instance id=2233, parent internal  id=692, parent id=WebsiteScaffoldingOctopus, description: "svn:  http://svn.corp.skyscanner.local/svn/SkyscannerWebsite/SEO"}, due to  error: Cannot build patch: java.lang.NullPointerException


[...]

[11:25:47]Patch is broken, can be found in file: G:\TeamCity\BuildAgent\temp\globalTmp\temp1067199266517313017patch_3485033
[11:25:47]Failed  to build patch for build #1.0.64219.1122 {build id=3485033,  buildTypeId=CommonUserExperience_WebApplications_WebsiteScaffoldingOctopus_11BuildAndPackage},  VCS root: "Website Scaffolding" {instance id=2233, parent internal  id=692, parent id=WebsiteScaffoldingOctopus, description: "svn:  http://svn.corp.skyscanner.local/svn/SkyscannerWebsite/SEO"}, due to  error: Cannot build patch: java.lang.NullPointerException

[11:25:47]
jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl$1:  Server was not able to build correct patch, most likely due to VCS  errors      at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.throwError(PatchDownloaderImpl.java:118)      at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatch(PatchDownloaderImpl.java:108)      at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.copyPatchAndCheck(PatchDownloaderImpl.java:69)      at jetbrains.buildServer.agent.impl.patch.UpdateSourcesPatcherBase.copyPatchToTempFile(UpdateSourcesPatcherBase.java:71)      at jetbrains.buildServer.agent.impl.patch.UpdateSourcesFromServer.updateSources(UpdateSourcesFromServer.java:60)      at jetbrains.buildServer.agent.impl.patch.UpdateSourcesBuildStageBase.doSourceUpdate(UpdateSourcesBuildStageBase.java:91)      at jetbrains.buildServer.agent.impl.patch.UpdateSourcesBuildStageBase.doRecoverableStage(UpdateSourcesBuildStageBase.java:59)      at jetbrains.buildServer.agent.impl.buildStages.startStages.RecoverableBuildStage.doLastAttempt(RecoverableBuildStage.java:112)      at jetbrains.buildServer.agent.impl.buildStages.startStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:70)      at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)      at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)      at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)      at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)      at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)      at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:70)      at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:50)      at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:266)      at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:51)      at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:231)      at java.lang.Thread.run(Thread.java:744) Caused by: jetbrains.buildServer.vcs.patches.UnsuccessfulPatchException:  Failed to build patch for build #1.0.64219.1122 {build id=3485033,  buildTypeId=CommonUserExperience_WebApplications_WebsiteScaffoldingOctopus_11BuildAndPackage},  VCS root: "Website Scaffolding" {instance id=2233, parent internal  id=692, parent id=WebsiteScaffoldingOctopus, description: "svn:  http://svn.corp.skyscanner.local/svn/SkyscannerWebsite/SEO"}, due to  error: Cannot build patch: java.lang.NullPointerException      at jetbrains.buildServer.vcs.patches.AbstractPatcher$1.fail(AbstractPatcher.java:93)      at jetbrains.buildServer.vcs.patches.LowLevelPatcher.readPatchStream(LowLevelPatcher.java:165)      at jetbrains.buildServer.vcs.patches.LowLevelPatcher.applyPatch(LowLevelPatcher.java:88)      at jetbrains.buildServer.vcs.patches.AbstractPatcher.applyPatch(AbstractPatcher.java:42)      at jetbrains.buildServer.agent.impl.patch.PatchApplierImpl.applyPatch(PatchApplierImpl.java:18)      at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatchInFileIsCompleted(PatchDownloaderImpl.java:88)      at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatch(PatchDownloaderImpl.java:98)

To fix this issue, we have tried :

  • Clean checkout directory (from the custom build interface)                                                                             - no effect
  • Clean checkout (using the Actions -> Enforce Clean Checkout)                                                                      - temporarily fixed the issues, usually for a few builds, then consistently broken again on all agents
  • Clean caches : vcsContentCache, svn, git, then agentFiles (from the Administration -> Diagnostics interface)     - no effect
  • Stop server service, delete contents of %TC_DATA_DIR%\system\caches, start service                                    - no effect
  • Systemally delete all workspaces for all Windows agents (Linux agents were not affected by this issue)             - limited effect, still seeing this issue on over half of the build configurations
  • Updated the jre to 7u75 (x64)                                                                                                                        - no effect


    Before upgrading, we have succesfully tested the upgrade process on a full clone of the production server, skipping only the indexing to save time.The fact that a simple reindexing has caused so much trouble and there wasn't even a hint about it in the upgrade documentation is very concerning and the discussion I started about making the TeamCity upgrade a regular occurence has now shifted towards decentralisation and if TeamCity is becoming a single point of failure for our CD model.

    0
    Comment actions Permalink

    Hi Paul,

    Most likely this is the following issue, caused by incorrect/problematic upgrade: https://youtrack.jetbrains.com/issue/TW-39810
    (copied from email thread)

    0
    Comment actions Permalink

    Thank you very much for pointing me to that bug report. Deleting svnkit-1.8.4.jar, svnkit-cli-1.8.4.jar and svnkit-javahl16-1.8.4.jar solved the problem.

    0

    Please sign in to leave a comment.