VCS gets stuck forever when attempting to checkout from SVN
During a build, when attempting to checkout from an svn server (running behind apache) the process seems to get stuck forever without actually progressing.
I can correctly see the SVN commit history via the TeamCity interface, so TeamCity is able to connect to the SVN server.
However, during the "Updating Sources" stage, it gets stuck forever with the following build log (I have hidden the actual SVN location by replacing with XXXXXXX):
[18:40:17]Finalize build settings
[18:40:22]Collecting changes in 1 VCS root
[18:40:23][Collecting changes in 1 VCS root] VCS Root details
[18:40:23][VCS Root details] "Trunk" {instance id=7, parent internal id=1, parent id=DbSteam_Trunk, description: "svn: XXXXXXXXXX"}
[18:40:23][Collecting changes in 1 VCS root] Compute revision for 'Trunk'
[18:40:23][Compute revision for 'Trunk'] Upper limit revision: 60674
[18:40:23][Compute revision for 'Trunk'] Latest commit attached to build configuration: 60674
[18:40:23][Compute revision for 'Trunk'] Computed revision: 60674
[18:40:23]Clearing temporary directory: C:\Gamedev\TeamCity\buildAgent\temp\buildTmp
[18:40:23]Publishing internal artifacts
[18:40:23][Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[18:40:23][Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[18:40:23]Checkout directory: C:\Gamedev\TeamCity\buildAgent\work\3f8c75a9ae4b7f54
[18:40:23]Updating sources: agent side checkout (running for 1m:04s)
[18:40:23][Updating sources] Using vcs information from agent file: 3f8c75a9ae4b7f54.xml
[18:40:23][Updating sources] VCS Root: Trunk (running for 1m:04s)
[18:40:23][VCS Root: Trunk] revision: 60674
[18:40:23][VCS Root: Trunk] Checkout from XXXXXXXXXXXXXX
[18:40:23][VCS Root: Trunk] Using working copy format 1.8
[18:40:23][VCS Root: Trunk] Reverting local changes for working directory
[18:40:23][Reverting local changes for working directory] svn revert in C:\Gamedev\TeamCity\buildAgent\work\3f8c75a9ae4b7f54
The actually checkout is being performed on the agent, and the ".svn" folder is being correctly created in the destination checkout directory. However, after waiting an hour, no files were actually checked out and the checkout directory remained at 33KB in size with only 3 files created in the ".svn" folder.
There does not appear to be any errors reported, and in the vcs-log all I can see is a constant stream of the following log:
[2016-08-16 18:16:47,054] DEBUG [ical executor 3] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:17:57,054] DEBUG [ical executor 4] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:19:07,054] DEBUG [ical executor 5] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:20:17,056] DEBUG [ical executor 1] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:21:27,059] DEBUG [ical executor 2] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:22:37,060] DEBUG [ical executor 3] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:23:47,062] DEBUG [ical executor 4] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:24:57,065] DEBUG [ical executor 5] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec [2016-08-16 18:26:07,067] DEBUG [ical executor 1] - uildServer.VCS.SVN.performance - getCurrentVersion start: 0 msec
Any help on this would be much appreciated!!!
Thanks
Please sign in to leave a comment.
Hi, regarding the native SVN support.
It is there, but without full SVN externals support. Please take a look at this ticket: https://youtrack.jetbrains.com/issue/TW-51254/Implement-native-SVN-checkout-support-instead-of-using-SVNKit
Hope this helps,
Hi Dann,
What TeamCity version do you use? If not the latest one (10.0.1 as of now), then please upgrade and check if the issue is reproduced.
Hi Alina,
Yes I am using the latest version 10.0.1.
Upon further investigation, it appears that it is actually the size of our SVN repository that is not working nicely with Team City.
Our repository is ~200GB in size, and it seems that in the BuildAgent temp directory, it is creating many .spool files, each around 500MB in size. This is causing the checkout process to appear like it is doing nothing for a very long time.
This is also creating a problem with regards to disk space requirements attempting to check out such a large repository.
Any suggestions?
Looks like it is part of SVNKIT to spool the entire command to disk first. Would be nice if there was a way to disable this (if one does not already exist that I have not yet found) from the TeamCity interface.
Hello Dann,
Do you have to checkout all 200Gb for the build? SVNKit really does spooling of HTTP response to the disk while checking out sources, and it cannot be disabled so far. The problem with "checkout process does nothing" is probably related to this issue: https://youtrack.jetbrains.com/issue/TW-11993
Why the spooling is the problem?
Regards,
Hi Kirill,
As you mentioned it would be alot nicer if there was some visual progress on the state of the checkout, especially when checking out large repositories which can take a long time.
I think that there is probably some disk overhead with using the SPOOLING method, as at peak I guess you will require double the repository size. This isn't a massive issues as generally the agents will have ample disk space but could be a problem for some setups.
More importantly though, I have found that using the built-in VCS system in Team City for SVN, which internally uses SVN-KIT seems to take considerably longer than a native SVN checkout.
We have decided to basically not use the VCS system in Team City at all for these reasons, and instead just add a manual build step that performs a checkout using the native client via the command line.
Hi Dann,
Thanks for the answer. We'll consider adding a native Svn checkout support in addition to Svnkit.
Kind regards,
Hi Kirill,
Is native SVN checkout supported now?
I encountered the same situation, since my repository is about 500GB now;
The svnkit checkouts very slow and after the first checkout, any updates will cause svn locked with following logs:
Checkout using SVNKit from *****
10:25:35 Using working copy format 1.8
10:25:35 Fast SVN update is enabled
10:25:35 Running 'svn cleanup' because of [Error during update from **** to **** svn: E155004: Working copy **** locked]
10:25:35 Failed to perform checkout on agent: svn cleanup failed: svn: E155004: Working copy **** locked
Kind regards