TeamCity fails with socket timeout when checking out large Git project

Answered

Permanently deleted user

Created February 13, 2017 13:09

Hi,

I'm integrating a fairly large Git project (du -sh .git = 471M) into TeamCity. It fails during the server side checkout:

Updating sources: server side checkout (running for 3m:49s)

[13:26:37][Updating sources] Using vcs information from server. Reason: no revision information for build configuration "Continuous Build" and checkout directory /almacenamiento/TeamCityAgent/work/18363b4dc9113f27 on agent

[13:26:37][Updating sources] Will perform clean checkout

[13:26:37][Updating sources] Clean checkout reasons

[13:26:37][Clean checkout reasons] Agent doesn't have any version of the project sources

[13:26:37][Clean checkout reasons] Checkout directory is empty or doesn't exist

[13:26:37][Updating sources] Transferring cached clean patch for VCS root: Web

[13:28:37][Updating sources] Transferring repository sources: 36.75 MB so far...

[13:29:58][Updating sources] Failed to build patch for build #4 {build id=1642, buildTypeId=BkoolWeb_ContinuousBuild}, VCS root: "Web" {instance id=53, parent internal id=8, parent id=BkoolWeb_Api, description: "http://${host}/web/bkool-web.git#refs/heads/master"}, due to error: Error while transferring cached patch

Looking at the teamcity-vcs.log this is due to a socket timeout:

[2017-02-13 13:19:38,981] WARN [rmal executor 1] - jetbrains.buildServer.VCS - Error while transferring cached patch /almacenamiento/BuildServer/system/caches/sources/8c18dd522db8b0b9.cache
[2017-02-13 13:19:38,983] WARN [rmal executor 1] - jetbrains.buildServer.VCS - Failed to build patch for build #3 {build id=1641, buildTypeId=BkoolWeb_ContinuousBuild}, VCS root: "Web" {instance id=53, parent internal id=8, parent id=BkoolWeb_Api, description: "http://${host}/web/bkool-web.git#refs/heads/master"}, due to error: Error while transferring cached patch
jetbrains.buildServer.vcs.VcsRootVcsException: Error while transferring cached patch
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatchForRoot(PatchComposer.java:90)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:18)
at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.buildPatch(BuildTypeImpl.java:126)
at jetbrains.buildServer.serverSide.impl.BuildStarter$3.call(BuildStarter.java:6)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:104)
at jetbrains.buildServer.serverSide.impl.BuildStarter$4.call(BuildStarter.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: jetbrains.buildServer.vcs.VcsException: Error while transferring cached patch
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.getOrMakeCacheableCleanPatch(PatchComposer.java:163)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.makeCleanPatch(PatchComposer.java:75)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.makePatch(PatchComposer.java:196)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatchForRoot(PatchComposer.java:51)
... 9 more
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:407)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:310)
at org.apache.catalina.connector.OutputBuffer.writeByte(OutputBuffer.java:451)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:77)
at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
at jetbrains.buildServer.serverSide.impl.ServerLoadMetricsAwareOutputStream.write(ServerLoadMetricsAwareOutputStream.java:9)
at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at jetbrains.buildServer.vcs.patches.OutputStreamPart.write(OutputStreamPart.java:48)
at jetbrains.buildServer.vcs.patches.PatchStreamUtil.writeBinary(PatchStreamUtil.java:76)
at jetbrains.buildServer.vcs.patches.LowLevelPatchTranslator$1.write(LowLevelPatchTranslator.java:70)
at jetbrains.buildServer.vcs.patches.LowLevelPatchBuilderImpl.changeBinary(LowLevelPatchBuilderImpl.java:67)
at jetbrains.buildServer.vcs.patches.LowLevelPatchBuilderImpl.changeBinary(LowLevelPatchBuilderImpl.java:32)
at jetbrains.buildServer.vcs.patches.LowLevelPatchBuilderWrapper.changeBinary(LowLevelPatchBuilderWrapper.java:65)
at jetbrains.buildServer.vcs.patches.LowLevelPatchBuilderWrapper.changeBinary(LowLevelPatchBuilderWrapper.java:26)
at jetbrains.buildServer.vcs.patches.LowLevelPatchTranslator.changeBinary(LowLevelPatchTranslator.java:78)
at jetbrains.buildServer.vcs.patches.LowLevelPatchTranslator.changeBinary(LowLevelPatchTranslator.java:29)
at jetbrains.buildServer.vcs.patches.LowLevelPatcher$2.apply(LowLevelPatcher.java:130)
at jetbrains.buildServer.vcs.patches.LowLevelPatcher.withContentRead(LowLevelPatcher.java:59)
at jetbrains.buildServer.vcs.patches.LowLevelPatcher.readPatchStream(LowLevelPatcher.java:128)
at jetbrains.buildServer.vcs.patches.LowLevelPatcher.applyPatch(LowLevelPatcher.java:104)
at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.getOrMakeCacheableCleanPatch(PatchComposer.java:179)
... 12 more
Caused by: java.net.SocketTimeoutException
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:174)

If I ssh to the machine where TeamCity's installed then I can do a git clone of the repo with no problem. The repo doesn't have any really big objects:

git verify-pack -v .git/objects/pack/pack-*.idx | sort -k 3 -n | tail -1

7786d97ce25501242119ccb3c119b2745f4cc105 blob 10914107 1364330 39728131

and in any case I've set the teamcity.git.stream.file.threshold.mb=256 in internal.properties.

And help/advice would be gratefully received,

Thanks

1 comment

Permanently deleted user

Created February 13, 2017 16:24

After upgrading to 10 the problem no longer occurs

Please sign in to leave a comment.