UsingTeamCity with very large Mercurial repos

I am evaluating TeamCity on Windows as a possible candidate for our new continuous integration server.  I have a very large Mercurial repo, the repo is about 10G and the source code that comes out is about 2G.  Creating an initial clone from the central hg repo on the local network takes 1-3 hours depending on traffic.  We can't even clone successfully over https, we use a Windows shared directory to pull and set up our hgrc with entries something like these:

default = \\hgserver.domain.net\repo

default-push = https://hgserver/repo

Getting the first clone and checkout done in TeamCity is proving problematic.  Repeated attempts timed out after about an hour.  Using this thread as a base: http://devnet.jetbrains.net/message/5218025#5218025

I was able to manually create the repo in the .BuildServer\system\caches\mercurial\hg_-4798151147468214717 directory.  Now it's creating the initial patch, not sure if that will succeed or timeout.

So, some questions:

1) Am I going to be able to get this to work or is this hg repo just too large?

2) Is there a series of steps I can perform to get the initial  repo, patch, and source checkout done manually without it taking so long in TeamCity?

3) I've seen references to Git timeout variables.  What timout variables are avaiable for me to modify in Mercurial and TeamCity?

4) TeamCity puts the repo in one dir, the source files in another, and I see references to temp dirs in the build file.  Where can I find a detailed reference of steps and dirs created/used in a build that has an hg pull/update and an MSbuild step?

Thanks,

Scott

6 comments
Comment actions Permalink

Hi Scott,

you can set internal property 'teamcity.hg.pull.timeout.seconds' to change pull timeout, its default value is 3600 (1 hour) that is why you get errors. This property was added in TeamCity 6.5.5 and our documentation (http://confluence.jetbrains.net/display/TCD65/Mercurial) didn't reflect this fact, I've updated it.

I suggest you to set this property to 14400 (4 hours), after that TeamCity should be able to make initial clone, and subsequent operations should be much faster than initial clone.

Answering to your questions:

> 1) Am I going to be able to get this to work or is this hg repo just too large?

In our installation we dial with 3Gb (git) repositories, so I think 10Gb repository should also work. But could you please explain why it is so big? Do you store binaries there? If so, you can benefit from mercurial extensions like largefiles (http://mercurial.selenic.com/wiki/LargefilesExtension).

> 2) Is there a series of steps I can perform to get the initial repo, patch, and source checkout done manually without it taking so long in TeamCity?

You can do all of these steps in your build script and TeamCity will not do any vcs-related operations. But usually, VCS operations are done by TeamCity, not by build script. You can find description of build lifecycle here: http://confluence.jetbrains.net/display/TCD65/Getting+Started#GettingStarted-BuildLifecycleinTeamCity

> 3) I've seen references to Git timeout variables. What timout variables are avaiable for me to modify in Mercurial and TeamCity?

All available variables are described here: http://confluence.jetbrains.net/display/TCD65/Mercurial.

> 4) TeamCity puts the repo in one dir, the source files in another, and I see references to temp dirs in the build file. Where can I find a detailed reference of steps and dirs created/used in a build that has an hg pull/update and an MSbuild step?

TeamCity transfers source code to the agent for you, so you don't need to do hg pull/update in your build script.
Since your repository is so big I suggest you to use an agent-side checkout with 'teamcity.hg.use.local.mirrors' option enabled. It works like following:

- on TeamCity server we have bare clone of your repository, we sync it periodically and use it to get information about new changes, it is not used for building patches.

- each build agent has its own bare clone of your repository (so called local mirror) which is updated every time build runs on that agent. Update of local mirror usually fast since only small amount of data is transferred.

- after local mirror is updated, we clone sources from this local mirror to the working directory of the build. Since clone is made on the same machine without any network interaction it is relatively fast (some improvements are planned here http://youtrack.jetbrains.com/issue/TW-19249).

- when sources are ready, TeamCity runs your build script, and there is no need to execute hg pull/update in it.

0
Comment actions Permalink

Thanks very much, this is all very helpful and I think it will get me past the initial setup problems.  

The repo has a few binaries but not really many.  There are also files that were deleted or moved elsewhere but the repo retains the historical change sets.  No one is willing to lose any history so we are still looking for ways to trim the repo down.  It is a big repo even without the files that have been culled already.

I'm out of the office but will be back in a few days and will try to implement your suggestions.

Thanks very much,
Scott

0
Comment actions Permalink

Okay, I think I'm half way there.

I added both values to buildAgent\conf\buildAgent.properties and I can see them reflected in the TC web interface under the build agent, Agent Parameters/Configuration Parameters.

I created the file .Buildserver\config\internal.properties and added a line to set teamcity.hg.pull.timeout.seconds but I don't see it reflected anywhere in the web interface and I'm still getting an hg timeout after an hour.  I even restarted both eht build agent and TeamCity services just to make sure the settings were read in.  What am I doing wrong?

Here's the build log from my last failed attempt:

[13:44:15]: Checking for changes
[13:44:17]: Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[13:44:17]: Checkout directory: C:\TCAgentCheckoutDir
[13:44:17]: Updating sources: agent side checkout... (1h:08m:26s)
[13:44:17]: [Updating sources: agent side checkout...] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[13:44:17]: [Updating sources: agent side checkout...] Cleaning C:\TCAgentCheckoutDir
[13:44:17]: [Updating sources: agent side checkout...] VCS Root: Test 1 VCS (1h:08m:26s)
[13:44:17]: [VCS Root: Test 1 VCS] revision: 46212:46af8de5ab24
[13:44:17]: [VCS Root: Test 1 VCS] Update local mirror at C:\TeamCity\buildAgent\system\mercurial\hg_-4798151147468214717
[13:44:17]: [VCS Root: Test 1 VCS] Start pulling changes from C:\TeamCity\buildAgent\system\mercurial\hg_-4798151147468214717
[13:52:42]: [VCS Root: Test 1 VCS] Local mirror changes successfully pulled
[13:52:42]: [VCS Root: Test 1 VCS] Init repository at C:\TCAgentCheckoutDir, remote repository is C:\TeamCity\buildAgent\system\mercurial\hg_-4798151147468214717
[13:52:43]: [VCS Root: Test 1 VCS] Start pulling changes from C:\TeamCity\buildAgent\system\mercurial\hg_-4798151147468214717
[14:52:44]: [Updating sources: agent side checkout...] Failed to perform checkout on agent: '"C:\Program Files\TortoiseHg\hg.exe" pull https://hgserver.mydoamain.net/repo' command failed. exception: Timeout exception
[14:52:45]: Publishing internal artifacts
[14:52:46]: [Publishing internal artifacts] Sending build.finish.properties.gz file
[14:52:45]: Build failed to start. Artifacts will not be published for this build
[14:52:46]: Build finished

0
Comment actions Permalink

do you use TeamCity 6.5.5 or later? I ask because teamcity.hg.pull.timeout.seconds was added in 6.5.5.

0
Comment actions Permalink

Also from you log I found that even though local mirror was created on the build agent, pull in the build directory was made from remote repository, not from local one. I created an issue for that: http://youtrack.jetbrains.com/issue/TW-19369. You can download hg-plugin with fix from here: http://teamcity.jetbrains.com/viewType.html?buildTypeId=bt242&tab=buildTypeStatusDiv,it is #snapshot-52. To install plugin save "mercurial.zip" from build's artifacts to the .BuildServer/plugins and restart the server.

0
Comment actions Permalink

Yes, I'm using 6.5.5.  I upgraded from (I think) 6.5.3 because your first message mentioned the timeout was added in 6.5.5.  I'll get the plugin to fix the remote pull issue.  Thanks.

Scott

0

Please sign in to leave a comment.