TeamCity 9.1.6 stuck on Updating source

Hi There,

We've been facing this issue on one of our git repositories. TeamCity gets stuck when creating git patch for a server-side checkout, the process times out after 1800 seconds on each try. We're not sure what's causing the issue.

The bare repository checked out in the cache/git folder is working well we tried using it as a remote and it worked fine so the issue doesn't seem to be related to git checkout.

Here's the log:

[09:59:11]Collecting changes in 2 VCS roots (8s)

[09:59:11][Collecting changes in 2 VCS roots] VCS Root details
[09:59:11][VCS Root details] "FaultyRepostiory" {instance id=42, parent internal id=2, parent id=Project_Apps_Services_TestApp, description: "https://repository.url/git/Project#refs/heads/work"}
[09:59:11][VCS Root details] "AnotherRepository" {instance id=27, parent internal id=6, parent id=MmpConnectors, description: "https://repository.url/git/AnotherProject#refs/heads/work"}
[09:59:20]Clearing temporary directory: C:\BuildAgent\temp\buildTmp
[09:59:20]Publishing internal artifacts
[09:59:20][Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[09:59:21][Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[09:59:20]Checkout directory: C:\BuildAgent\work\320a512a30bcd4da
[09:59:20]Updating sources: server side checkout (30m:19s)
[09:59:20][Updating sources] Using vcs information from server. Reason: no revision information for build configuration "Test Build Configuration" and checkout directory C:\BuildAgent\work\320a512a30bcd4da on agent
[09:59:21][Updating sources] Will perform clean checkout
[09:59:21][Updating sources] Clean checkout reasons
[09:59:21][Clean checkout reasons] Agent doesn't have any version of the project sources
[09:59:21][Clean checkout reasons] Checkout directory is empty or doesn't exist
[09:59:21][Updating sources] Building and caching clean patch for VCS root: FaultyRepostiory
[10:29:40]
[Updating sources] Failed to build patch for build #64 {build id=4284, buildTypeId=Project_TestBuildConfiguration}, VCS root: "FaultyRepostiory" {instance id=42, parent internal id=2, parent id=Project_Apps_Services_TestApp, description: "https://repository.url/git/Project#refs/heads/work"}, due to error: 'build patch' command failed.
exception: Timeout exception: the process did not produce output longer than 1800 seconds; Process: c:\TeamCity\jre\bin\java -Xmx1024M -cp C:/TeamCity/webapps/ROOT/WEB-INF/lib/common-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-logging-1.1.3.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/JavaEWAH-0.7.9.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/openapi.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/log4j-1.2.12.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/server-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/jdom.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpcore-4.3.2.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpclient-4.3.4.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/trove4j.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit.http.apache-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-common.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-codec-1.6.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-server.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/jsch-0.1.51.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/patches-impl.jar jetbrains.buildServer.buildTriggers.vcs.git.patch.GitPatchProcess https://repository.url/git/Project
exit code: -1
[10:29:40][Updating sources] Repository sources transferred
[10:29:40]Will repeat attempt when server will be available, number of attempts left: 2
[10:29:50]Updating sources: server side checkout (30m:18s)
[10:29:50][Updating sources] Will perform clean checkout
[10:29:50][Updating sources] Clean checkout reasons
[10:29:50][Clean checkout reasons] Agent doesn't have any version of the project sources
[10:29:50][Clean checkout reasons] Checkout directory is empty or doesn't exist
[10:29:50][Updating sources] Building and caching clean patch for VCS root: FaultyRepostiory
[11:00:09]
[Updating sources] Failed to build patch for build #64 {build id=4284, buildTypeId=Project_TestBuildConfiguration}, VCS root: "FaultyRepostiory" {instance id=42, parent internal id=2, parent id=Project_Apps_Services_TestApp, description: "https://repository.url/git/Project#refs/heads/work"}, due to error: 'build patch' command failed.
exception: Timeout exception: the process did not produce output longer than 1800 seconds; Process: c:\TeamCity\jre\bin\java -Xmx1024M -cp C:/TeamCity/webapps/ROOT/WEB-INF/lib/common-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-logging-1.1.3.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/JavaEWAH-0.7.9.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/openapi.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/log4j-1.2.12.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/server-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/jdom.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpcore-4.3.2.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpclient-4.3.4.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/trove4j.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit.http.apache-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-common.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-codec-1.6.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-server.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/jsch-0.1.51.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/patches-impl.jar jetbrains.buildServer.buildTriggers.vcs.git.patch.GitPatchProcess https://repository.url/git/Project
exit code: -1
[11:00:09][Updating sources] Repository sources transferred
[11:00:09]Will repeat attempt when server will be available, number of attempts left: 1
[11:00:19]Updating sources: server side checkout (30m:18s)
[11:00:19][Updating sources] Will perform clean checkout
[11:00:19][Updating sources] Clean checkout reasons
[11:00:19][Clean checkout reasons] Agent doesn't have any version of the project sources
[11:00:19][Clean checkout reasons] Checkout directory is empty or doesn't exist
[11:00:19][Updating sources] Building and caching clean patch for VCS root: FaultyRepostiory
[11:30:38]
[Updating sources] Failed to build patch for build #64 {build id=4284, buildTypeId=Project_TestBuildConfiguration}, VCS root: "FaultyRepostiory" {instance id=42, parent internal id=2, parent id=Project_Apps_Services_TestApp, description: "https://repository.url/git/Project#refs/heads/work"}, due to error: 'build patch' command failed.
exception: Timeout exception: the process did not produce output longer than 1800 seconds; Process: c:\TeamCity\jre\bin\java -Xmx1024M -cp C:/TeamCity/webapps/ROOT/WEB-INF/lib/common-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-logging-1.1.3.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/JavaEWAH-0.7.9.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/openapi.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/log4j-1.2.12.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/server-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/jdom.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpcore-4.3.2.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpclient-4.3.4.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/trove4j.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit.http.apache-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-common.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-codec-1.6.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-server.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/jsch-0.1.51.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/patches-impl.jar jetbrains.buildServer.buildTriggers.vcs.git.patch.GitPatchProcess https://repository.url/git/Project
exit code: -1
[11:30:38][Updating sources] Repository sources transferred
[11:30:38]Patch is broken, can be found in file: C:\BuildAgent\temp\buildTmp\temp6145510904653027380patch_4284
[11:30:38]
Failed to build patch for build #64 {build id=4284, buildTypeId=Project_TestBuildConfiguration}, VCS root: "FaultyRepostiory" {instance id=42, parent internal id=2, parent id=Project_Apps_Services_TestApp, description: "https://repository.url/git/Project#refs/heads/work"}, due to error: 'build patch' command failed.
exception: Timeout exception: the process did not produce output longer than 1800 seconds; Process: c:\TeamCity\jre\bin\java -Xmx1024M -cp C:/TeamCity/webapps/ROOT/WEB-INF/lib/common-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-logging-1.1.3.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/JavaEWAH-0.7.9.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/openapi.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/log4j-1.2.12.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/server-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/jdom.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpcore-4.3.2.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpclient-4.3.4.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/trove4j.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit.http.apache-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-common.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-codec-1.6.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-server.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/jsch-0.1.51.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/patches-impl.jar jetbrains.buildServer.buildTriggers.vcs.git.patch.GitPatchProcess https://repository.url/git/Project
exit code: -1
[11:30:38]
jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl$1: Server was not able to build correct patch, most likely due to VCS errors
at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.throwError(PatchDownloaderImpl.java:119)
at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatch(PatchDownloaderImpl.java:109)
at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.copyPatchAndCheck(PatchDownloaderImpl.java:70)
at jetbrains.buildServer.agent.impl.patch.UpdateSourcesPatcherBase.copyPatchToTempFile(UpdateSourcesPatcherBase.java:71)
at jetbrains.buildServer.agent.impl.patch.UpdateSourcesFromServer.updateSources(UpdateSourcesFromServer.java:57)
at jetbrains.buildServer.agent.impl.patch.UpdateSourcesBuildStageBase.doSourceUpdate(UpdateSourcesBuildStageBase.java:91)
at jetbrains.buildServer.agent.impl.patch.UpdateSourcesBuildStageBase.doRecoverableStage(UpdateSourcesBuildStageBase.java:59)
at jetbrains.buildServer.agent.impl.buildStages.startStages.RecoverableBuildStage.doLastAttempt(RecoverableBuildStage.java:112)
at jetbrains.buildServer.agent.impl.buildStages.startStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:70)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:31)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor$1.callStage(BuildStagesExecutor.java:24)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.callRunStage(StagesExecutor.java:78)
at jetbrains.buildServer.agent.impl.buildStages.StagesExecutor.doStages(StagesExecutor.java:37)
at jetbrains.buildServer.agent.impl.buildStages.BuildStagesExecutor.doStages(BuildStagesExecutor.java:24)
at jetbrains.buildServer.agent.impl.BuildRunActionImpl.doStages(BuildRunActionImpl.java:70)
at jetbrains.buildServer.agent.impl.BuildRunActionImpl.runBuild(BuildRunActionImpl.java:50)
at jetbrains.buildServer.agent.impl.BuildAgentImpl.doActualBuild(BuildAgentImpl.java:288)
at jetbrains.buildServer.agent.impl.BuildAgentImpl.access$100(BuildAgentImpl.java:53)
at jetbrains.buildServer.agent.impl.BuildAgentImpl$1.run(BuildAgentImpl.java:252)
at java.lang.Thread.run(Thread.java:744)
Caused by: jetbrains.buildServer.vcs.patches.UnsuccessfulPatchException: Failed to build patch for build #64 {build id=4284, buildTypeId=Project_TestBuildConfiguration}, VCS root: "FaultyRepostiory" {instance id=42, parent internal id=2, parent id=Project_Apps_Services_TestApp, description: "https://repository.url/git/Project#refs/heads/work"}, due to error: 'build patch' command failed.
exception: Timeout exception: the process did not produce output longer than 1800 seconds; Process: c:\TeamCity\jre\bin\java -Xmx1024M -cp C:/TeamCity/webapps/ROOT/WEB-INF/lib/common-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-logging-1.1.3.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/JavaEWAH-0.7.9.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/openapi.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/log4j-1.2.12.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/server-api.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/jdom.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpcore-4.3.2.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/httpclient-4.3.4.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/trove4j.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/slf4j-log4j12-1.7.5.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit.http.apache-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-common.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/commons-codec-1.6.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/git-server.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/jsch-0.1.51.jar;C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/jetbrains.git/server/org.eclipse.jgit-3.7.0.201502260915-r.jar;C:/TeamCity/webapps/ROOT/WEB-INF/lib/patches-impl.jar jetbrains.buildServer.buildTriggers.vcs.git.patch.GitPatchProcess https://repository.url/git/Project
exit code: -1
at jetbrains.buildServer.vcs.patches.AbstractPatcher$1.fail(AbstractPatcher.java:93)
at jetbrains.buildServer.vcs.patches.LowLevelPatcher.readPatchStream(LowLevelPatcher.java:181)
at jetbrains.buildServer.vcs.patches.LowLevelPatcher.applyPatch(LowLevelPatcher.java:104)
at jetbrains.buildServer.vcs.patches.AbstractPatcher.applyPatch(AbstractPatcher.java:42)
at jetbrains.buildServer.agent.impl.patch.PatchApplierImpl.applyPatch(PatchApplierImpl.java:18)
at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatchInFileIsCompleted(PatchDownloaderImpl.java:89)
at jetbrains.buildServer.agent.impl.patch.PatchDownloaderImpl.checkPatch(PatchDownloaderImpl.java:99)
... 18 more
[11:30:38]Publishing internal artifacts
[11:30:38][Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[11:30:38][Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[11:30:38]Build failed to start. Artifacts will not be published for this build
[11:30:39]Build finished

There are no thread dumps present in the repository monitoring folder. The server VCS log also doesn't show anything other than what's pasted above.

Could someone please help?