TeamCity getting stuck at “Updating sources” on one Git repo
I've just set up TeamCity 8.0 on Windows to build the projects (Visual Studio solutions) in four separate Git repositories. It is working as expected on three of them, but on one it gets stuck at Updating sources. The settings for the four VCS roots are identical (apart from the Git repo fetch URLs).
All of the TeamCity projects are using server-side checkout.
The build log for the problematic project contains
[15:36:34]: bt1 (running for 41m:38s)
[15:36:34]: Checking for changes (running for 41m:37s)
[15:36:37]: Publishing internal artifacts
[15:36:38]: [Publishing internal artifacts] Sending build.start.properties.gz file
[15:36:37]: Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[15:36:37]: Checkout directory: C:\TeamCity\buildAgent\work\62d0281b7178c739
[15:36:37]: Updating sources: server side checkout (running for 41m:34s)
[15:36:38]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[15:36:38]: [Updating sources] Building and caching clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
Using Process Monitor on the build agent I can see that it is hammering the following folder with ReadFile operations.
C:\ProgramData\JetBrains\TeamCity\system\caches\git\git-11F9493A.git\objects\pack\pack-1490ccc8f7896ab876413465c4b48e87448bed35.pack
That .pack file is about 300MB in size, which I believe to be the same size as the Git repo.
I've tried deleting this folder and re-starting the TeamCity build agent and build server Windows services, but it just get re-created and then TeamCity hammers it once more.
Are there any other recommendations for how to troubleshoot this kind of issue?
(Question also posted on StackOverflow)
All of the TeamCity projects are using server-side checkout.
The build log for the problematic project contains
[15:36:34]: bt1 (running for 41m:38s)
[15:36:34]: Checking for changes (running for 41m:37s)
[15:36:37]: Publishing internal artifacts
[15:36:38]: [Publishing internal artifacts] Sending build.start.properties.gz file
[15:36:37]: Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[15:36:37]: Checkout directory: C:\TeamCity\buildAgent\work\62d0281b7178c739
[15:36:37]: Updating sources: server side checkout (running for 41m:34s)
[15:36:38]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[15:36:38]: [Updating sources] Building and caching clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
Using Process Monitor on the build agent I can see that it is hammering the following folder with ReadFile operations.
C:\ProgramData\JetBrains\TeamCity\system\caches\git\git-11F9493A.git\objects\pack\pack-1490ccc8f7896ab876413465c4b48e87448bed35.pack
That .pack file is about 300MB in size, which I believe to be the same size as the Git repo.
I've tried deleting this folder and re-starting the TeamCity build agent and build server Windows services, but it just get re-created and then TeamCity hammers it once more.
Are there any other recommendations for how to troubleshoot this kind of issue?
(Question also posted on StackOverflow)
Please sign in to leave a comment.
Hi Richard,
in order to collect changes TeamCity needs to have a clone of git repository on TeamCity server machine. The directory you mentioned - C:\ProgramData\JetBrains\TeamCity\system\caches\git\git-11F9493A.git - is a bare clone of the git VCS root you use in this configuration. Initial clone of git repository can take some time. In your case it takes almost 40 minutes which seems to be quite a lot for 300Mb repository, do you have a slow network connection? What time does it takes to do a clone from command line?
Hi Dimitry,
Does this help diagnose the issue?Thanks for your reply.
It only takes a couple of minutes to clone my repo from the command line; I don't have a slow network connection.
I left this build running over the weekend, and now have a more interesting error in the logs.
[18:20:12]: bt8 (36h:0m:18s)[18:20:12]: Checking for changes (running for 36h:0m:18s)
[18:20:15]: Publishing internal artifacts
[18:20:15]: [Publishing internal artifacts] Sending build.start.properties.gz file
[18:20:15]: Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[18:20:15]: Checkout directory: C:\TeamCity\buildAgent\work\bb4af4f56f498d4
[18:20:15]: Updating sources: server side checkout (11h:59m:58s)
[18:20:15]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[18:20:15]: [Updating sources] Building and caching clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
[06:20:13]: [Updating sources] Problem while loading patch data stream: Read timed out
[06:20:13]: Will repeat attempt when server will be available, number of attempts left: 2
[06:20:23]: Updating sources: server side checkout (11h:59m:58s)
[06:20:23]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[06:20:23]: [Updating sources] Building and caching clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
[18:20:22]: [Updating sources] Problem while loading patch data stream: Read timed out
[18:20:22]: Will repeat attempt when server will be available, number of attempts left: 1
[18:20:32]: Updating sources: server side checkout (11h:59m:58s)
[18:20:32]: [Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[05:50:32]: [Updating sources] Transferring cached clean patch for VCS root: git@qa.company.com:Company.WebSite.git#master
[05:50:32]: [Updating sources] Failed to build patch for build #1 {build id=49}, VCS root: "git@qa.company.com:Company.WebSite.git#master" {instance id=11, parent internal id=11, parent id=CompanyWebsite_GitQaCompanyComCompanyWebSiteGitMaster, description: "git@qa.company.com:Company.WebSite.git#master"}, due to error: Cannot build patch: ClientAbortException: java.io.IOException: An established connection was aborted by the software in your host machine
[05:50:32]: [Updating sources] Failed to build patch for build #1 {build id=49}, due to error: null
[06:20:31]: [Updating sources] Problem while loading patch data stream: Read timed out
[06:20:31]: Problem while loading patch data stream: Read timed out
[06:20:31]: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)(remaining call stack removed...)
Do you have large binaries in your repository? If so, it is possible that you are facing http://youtrack.jetbrains.com/issue/TW-14947. To workaround it set an internal parameter teamcity.git.stream.file.threshold.mb to the size of the biggest object in your repository in megabytes, like following
teamcity.git.stream.file.threshold.mb=256
Default value is 128, maybe it is not enough in your case. Let me know if it helps.
Hmm.. that's strange, 4mb files shouldn't cause such a slow checkout. Could you please check if your repository had large files in the past? To do that first run the following command in your repository:
git verify-pack -v .git/objects/pack/pack-*.idx | sort -k 3 -n | tail -1
it prints the larges object in repository in a format:
SHA1 type size size-in-pack-file offset-in-packfile
This command could take some time, on my 3Gb repository it took several minutes.
To find the path of largest file run:
git rev-list --objects --all | grep <SHA1 of the largest object>
Please provide an output of both commands.
(sorry about the messed up formatting - newlines in my messages are getting deleted when I post)
So it says that "DDL/SQL Server/PROD/CompanyWebsite schema + data.sql" is 154mb, do you still have this file in the repo? If you don't it seems like even old large files can cause problems..
Unless this file was changed recently, even old large files can cause the problem with slow checkout. Thanks for your help!
We are experiencing a similar issue with TeamCity 8.0.5. However, we're using Mercurial, not git. Is there another internal properties variable that we should set for Mercurial?
Hi Jerry,
I don't think mercurial has this kind of issue. Please provide more details on the problem you are facing.