VCS gets stuck forever when attempting to checkout from SVN

During a build, when attempting to checkout from an svn server (running behind apache) the process seems to get stuck forever without actually progressing.

 

I can correctly see the SVN commit history via the TeamCity interface, so TeamCity is able to connect to the SVN server.

However, during the "Updating Sources" stage, it gets stuck forever with the following build log (I have hidden the actual SVN location by replacing with XXXXXXX):

 

[18:40:17]Finalize build settings
[18:40:22]Collecting changes in 1 VCS root
[18:40:23][Collecting changes in 1 VCS root] VCS Root details
[18:40:23][VCS Root details] "Trunk" {instance id=7, parent internal id=1, parent id=DbSteam_Trunk, description: "svn: XXXXXXXXXX"}
[18:40:23][Collecting changes in 1 VCS root] Compute revision for 'Trunk'
[18:40:23][Compute revision for 'Trunk'] Upper limit revision: 60674
[18:40:23][Compute revision for 'Trunk'] Latest commit attached to build configuration: 60674
[18:40:23][Compute revision for 'Trunk'] Computed revision: 60674
[18:40:23]Clearing temporary directory: C:\Gamedev\TeamCity\buildAgent\temp\buildTmp
[18:40:23]Publishing internal artifacts
[18:40:23][Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[18:40:23][Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[18:40:23]Checkout directory: C:\Gamedev\TeamCity\buildAgent\work\3f8c75a9ae4b7f54
[18:40:23]Updating sources: agent side checkout (running for 1m:04s)
[18:40:23][Updating sources] Using vcs information from agent file: 3f8c75a9ae4b7f54.xml
[18:40:23][Updating sources] VCS Root: Trunk (running for 1m:04s)
[18:40:23][VCS Root: Trunk] revision: 60674
[18:40:23][VCS Root: Trunk] Checkout from XXXXXXXXXXXXXX
[18:40:23][VCS Root: Trunk] Using working copy format 1.8
[18:40:23][VCS Root: Trunk] Reverting local changes for working directory
[18:40:23][Reverting local changes for working directory] svn revert in C:\Gamedev\TeamCity\buildAgent\work\3f8c75a9ae4b7f54
 
 
The actually checkout is being performed on the agent, and the ".svn" folder is being correctly created in the destination checkout directory. However, after waiting an hour, no files were actually checked out and the checkout directory remained at 33KB in size with only 3 files created in the ".svn" folder.
 
There does not appear to be any errors reported, and in the vcs-log all I can see is a constant stream of the following log:
 
[2016-08-16 18:16:47,054]  DEBUG [ical executor 3] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:17:57,054]  DEBUG [ical executor 4] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:19:07,054]  DEBUG [ical executor 5] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:20:17,056]  DEBUG [ical executor 1] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:21:27,059]  DEBUG [ical executor 2] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:22:37,060]  DEBUG [ical executor 3] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:23:47,062]  DEBUG [ical executor 4] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:24:57,065]  DEBUG [ical executor 5] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 
[2016-08-16 18:26:07,067]  DEBUG [ical executor 1] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec 


Any help on this would be much appreciated!!!

Thanks
7 comments

Hi Dann,

What TeamCity version do you use? If not the latest one (10.0.1 as of now), then please upgrade and check if the issue is reproduced.

0

Hi Alina,

Yes I am using the latest version 10.0.1.

0

Upon further investigation, it appears that it is actually the size of our SVN repository that is not working nicely with Team City.

Our repository is ~200GB in size, and it seems that in the BuildAgent temp directory, it is creating many .spool files, each around 500MB in size. This is causing the checkout process to appear like it is doing nothing for a very long time. 

This is also creating a problem with regards to disk space requirements attempting to check out such a large repository.

Any suggestions?

0

Looks like it is part of SVNKIT to spool the entire command to disk first. Would be nice if there was a way to disable this (if one does not already exist that I have not yet found) from the TeamCity interface.

0

Hello Dann,

   Do you have to checkout all 200Gb for the build? SVNKit really does spooling of HTTP response to the disk while checking out sources, and it cannot be disabled so far. The problem with "checkout process does nothing" is probably related to this issue: https://youtrack.jetbrains.com/issue/TW-11993

   Why the spooling is the problem?

   Regards,

0

Hi Kirill,

As you mentioned it would be alot nicer if there was some visual progress on the state of the checkout, especially when checking out large repositories which can take a long time.

I think that there is probably some disk overhead with using the SPOOLING method, as at peak I guess you will require double the repository size. This isn't a massive issues as generally the agents will have ample disk space but could be a problem for some setups.

More importantly though, I have found that using the built-in VCS system in Team City for SVN, which internally uses SVN-KIT seems to take considerably longer than a native SVN checkout.

We have decided to basically not use the VCS system in Team City at all for these reasons, and instead just add a manual build step that performs a checkout using the native client via the command line.

 

0

Hi Dann,

   Thanks for the answer. We'll consider adding a native Svn checkout support in addition to Svnkit.

   Kind regards,  

0

Please sign in to leave a comment.