TeamCity getting stuck at “Updating sources” on one Git repo

I've just set up TeamCity 8.0 on Windows to build the projects (Visual Studio solutions) in four separate Git repositories. It is working as expected on three of them, but on one it gets stuck at Updating sources. The settings for the four VCS roots are identical (apart from the Git repo fetch URLs).

All of the TeamCity projects are using server-side checkout.

The build log for the problematic project contains

[15:36:34]: bt1 (running for 41m:38s)
[15:36:34]: Checking for changes (running for 41m:37s)
[15:36:37]: Publishing internal artifacts
[15:36:38]: [Publishing internal artifacts] Sending build.start.properties.gz file
[15:36:37]: Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[15:36:37]: Checkout directory: C:\TeamCity\buildAgent\work\62d0281b7178c739
[15:36:37]: Updating sources: server side checkout (running for 41m:34s)
[15:36:38]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[15:36:38]: [Updating sources] Building and caching clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master

Using Process Monitor on the build agent I can see that it is hammering the following folder with ReadFile operations.

C:\ProgramData\JetBrains\TeamCity\system\caches\git\git-11F9493A.git\objects\pack\pack-1490ccc8f7896ab876413465c4b48e87448bed35.pack

That .pack file is about 300MB in size, which I believe to be the same size as the Git repo.

I've tried deleting this folder and re-starting the TeamCity build agent and build server Windows services, but it just get re-created and then TeamCity hammers it once more.

Are there any other recommendations for how to troubleshoot this kind of issue?

(Question also posted on StackOverflow)
12 comments
Comment actions Permalink

Hi Richard,

in order to collect changes TeamCity needs to have a clone of git repository on TeamCity server machine. The directory you mentioned - C:\ProgramData\JetBrains\TeamCity\system\caches\git\git-11F9493A.git - is a bare clone of the git VCS root you use in this configuration. Initial clone of git repository can take some time. In your case it takes almost 40 minutes which seems to be quite a lot for 300Mb repository, do you have a slow network connection? What time does it takes to do a clone from command line?  

0
Comment actions Permalink

Hi Dimitry,


Thanks for your reply.


It only takes a couple of minutes to clone my repo from the command line; I don't have a slow network connection.


I left this build running over the weekend, and now have a more interesting error in the logs.

Does this help diagnose the issue?


[18:20:12]: bt8 (36h:0m:18s)
[18:20:12]: Checking for changes (running for 36h:0m:18s)
[18:20:15]: Publishing internal artifacts
[18:20:15]:  [Publishing internal artifacts] Sending build.start.properties.gz file
[18:20:15]: Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[18:20:15]: Checkout directory: C:\TeamCity\buildAgent\work\bb4af4f56f498d4
[18:20:15]: Updating sources: server side checkout (11h:59m:58s)
[18:20:15]:  [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[18:20:15]:  [Updating sources] Building and caching clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
[06:20:13]:  [Updating sources] Problem while loading patch data stream: Read timed out
[06:20:13]: Will repeat attempt when server will be available, number of attempts left: 2
[06:20:23]: Updating sources: server side checkout (11h:59m:58s)
[06:20:23]:  [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[06:20:23]:  [Updating sources] Building and caching clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
[18:20:22]:  [Updating sources] Problem while loading patch data stream: Read timed out
[18:20:22]: Will repeat attempt when server will be available, number of attempts left: 1
[18:20:32]: Updating sources: server side checkout (11h:59m:58s)
[18:20:32]:  [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[05:50:32]:  [Updating sources] Transferring cached clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
[05:50:32]:  [Updating sources] Failed to build patch for build #1 {build id=49}, VCS root: "git@qa.company.com:Company.WebSite.git#master" {instance id=11, parent internal id=11, parent id=CompanyWebsite_GitQaCompanyComCompanyWebSiteGitMaster, description: "git@qa.company.com:Company.WebSite.git#master"}, due to error: Cannot build patch: ClientAbortException:  java.io.IOException: An established connection was aborted by the software in your host machine
[05:50:32]:  [Updating sources] Failed to build patch for build #1 {build id=49}, due to error: null
[06:20:31]:  [Updating sources] Problem while loading patch data stream: Read timed out
[06:20:31]: Problem while loading patch data stream: Read timed out
[06:20:31]: java.net.SocketTimeoutException: Read timed out
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(Unknown Source)

(remaining call stack removed...)
0
Comment actions Permalink

Do you have large binaries in your repository? If so, it is possible that you are facing http://youtrack.jetbrains.com/issue/TW-14947. To workaround it set an internal parameter teamcity.git.stream.file.threshold.mb to the size of the biggest object in your repository in megabytes, like following

teamcity.git.stream.file.threshold.mb=256

Default value is 128, maybe it is not enough in your case. Let me know if it helps.

0
Comment actions Permalink
Success!The largest file we have in our repository is just under 4MB, however I set the parameter as requested (I had to create a new internal.properties file as one did not already exist).The build completed in just a few minutes.Thanks for the help.
0
Comment actions Permalink

Hmm.. that's strange, 4mb files shouldn't cause such a slow checkout. Could you please check if your repository had large files in the past? To do that first run the following command in your repository:

git verify-pack -v .git/objects/pack/pack-*.idx | sort -k 3 -n | tail -1

it prints the larges object in repository in a format:

SHA1 type size size-in-pack-file offset-in-packfile

This command could take some time, on my 3Gb repository it took several minutes.

To find the path of largest file run:

git rev-list --objects --all | grep <SHA1 of the largest object>

Please provide an output of both commands.

0
Comment actions Permalink
Sure, here are the results of running those two commands:$ git verify-pack -v .git/objects/pack/pack-*.idx | sort -k 3 -n | tail -1351daa61e715bf6c2782e41d8eb9e05ddf4b6c1c blob   154375992 22735012 1436980$ git rev-list --objects --all | grep 351daa61e715bf6c2782e41d8eb9e05ddf4b6c1c351daa61e715bf6c2782e41d8eb9e05ddf4b6c1c DDL/SQL Server/PROD/CompanyWebsite schema + data.sql
0
Comment actions Permalink

(sorry about the messed up formatting - newlines in my messages are getting deleted when I post)

0
Comment actions Permalink

So it says that "DDL/SQL Server/PROD/CompanyWebsite schema + data.sql" is 154mb, do you still have this file in the repo? If you don't it seems like even old large files can cause problems..

0
Comment actions Permalink
Yes, that file does still exist in the repo, but it is currently 89MB in size.
0
Comment actions Permalink

Unless this file was changed recently, even old large files can cause the problem with slow checkout. Thanks for your help!

0
Comment actions Permalink

We are experiencing a similar issue with TeamCity 8.0.5.  However, we're using Mercurial, not git.  Is there another internal properties variable that we should set for Mercurial?

0
Comment actions Permalink

Hi Jerry,

I don't think mercurial has this kind of issue. Please provide more details on the problem you are facing.

0

Please sign in to leave a comment.