performance problems with TeamCity 4.0.2 and Subversion checkout

I recently installed (not upgraded) TC 4.0.2. Previously we'd been using TC 3.1. In 3.1 we had TC set up to clean sources prior to checkout from svn. The checkout of our full source tree took 2-3 minutes. In TC 4.0.2 with the same 'Clean all files before build' check box set, it takes TC 45-50 minutes to check out the same exact source code! If I do not check 'Clean all files before build' the first time a check out is done TC takes the same ridiculously long time to get the sources, but from then on it's using its new "fast update" -- from the log "[15:26:44]: Will use fast SVN update" -- and the checkout takes maybe a minute.

However, we want to use the 'Clean all files before build' option because of a separate issue with using TC and Maven2 build runner that I won't get into here.

So, why is TC 4.0.2 so much slower with svn checkouts then it was in TC 3.1? Has anyone else noticed this problem?

TC is configured to use the 'automatically on agent' checkout mode and we don't set the checkout directory so it uses the default agent directory.

Thanks.

14 comments
Comment actions Permalink

One additional piece of information...

I have TeamCity running as user 'teamcity'. If I check out the source from svn with the same svn URL by hand from the shell the checkout times are back to what I'd expect for a fresh checkout ... about 3 minutes give or take, for our source tree. So it seems to me to be something about TC rather than our svn server or the URL.

0
Comment actions Permalink

Hello Ray,

   I'll try to investigate the problem locally. Could you please tell the rough number of files in your project and total size of the project?
   I'd like to reproduce your situation locally.

   Could you please also enable debug information and attach logs from buildAgent/log directory?

   Thanks!
   KIR

0
Comment actions Permalink

We have about 18500 files, of which about 8000 are directories. The size is


% du -sk test
193980  test



 

Here are some statistics. When I do it from the command line I get ...

% time svn co svn+ssh://www.financeit.com/svn/fidex/trunk/fidex
<snip>

3.288u 3.166s 1:20.12 8.0%      0+0k 3240+374432io 27pf+0w


When TeamCity does it with the same svn URL, from the build log ...

[07:50:58]: Checking /home/teamcity/TeamCity/buildAgent/work/e8992470a2e40ab3 out from svn
[07:50:58]: Will use fast SVN update
[08:41:47]: MAVEN_OPTS =  -Xmx768m <snip>


It took 51 minutes! This is with the 'clear sources' checked.

I have attached the build logs. Unfortunately the log is set to roll, and it rolled 3 times. But the config is set to only keep 1 old log file, so the log file from the start of the check out is gone. I don't know if you need that or not. If you do let me know and I can try and grab that for you.

As I mentioned, it worked great in TC3 when checking out the full source tree every time, so something broke in TC4. Other than that I really like TC




Attachment(s):
svnlog.zip
0
Comment actions Permalink

Hello Ray,

   Is there difference in checkout times when DEBUG logging is enabled and when it is disabled?
   There is a noticeable impact of SVN debug logging on performance.

   Kind regards,
   KIR

0
Comment actions Permalink

Hello Ray,

  Could you please make another test.
  On the build agent, please download and unpack org.tmatesoft.svn_1.2.3.5521.standalone.zip .
  In the directory, there is a jsvn script. Please setup JAVA_HOME environment variable and try to checkout sources using this script
  (it accepts the same parameters as svn).
  Something like
  JAVA_HOME=/path/java time jsvn co svn+ssh://www.financeit.com/svn/fidex/trunk/fidex

  How much time would it take?

  Kind regards,
  KIR

0
Comment actions Permalink

No, the debug logging didn't have much of an impact. It was definitely still taking 45-50 minutes in TC vs 2 minutes by hand.

0
Comment actions Permalink

Kirill, when I try and download the file, I get a login prompt from a TeamCity server at that site. Perhaps you can just attach the zip to a post? Thanks.

0
Comment actions Permalink

Ray,

  There is a "Login as guest" link at the bottom of the page - you can use it.

  Regards,
  KIR

0
Comment actions Permalink

The problem definitely seems to be the svnkit implementation ... perhaps specific to svn+ssh.


% time svnkit-1.2.3.5521/jsvn co svn+ssh://www.financeit.com/svn/fidex/trunk/fidex
20.574u 5.278s 53:51.01 0.7%  0+0k 96+357448io 1pf+0w


Did TC3 use a different svn implementation?


0
Comment actions Permalink

Hello Ray,

  TeamCity 3.1.2 uses SVNKit/1.2.0-beta4 . TeamCity 3.1.1 uses earlier version.
  TeamCity 4.0.2 uses SVNKit/1.2.1 r5297

  Which 3.x build of TeamCity did you have installed? Given that the problem relates to SVNKit, I'm going to inform developers of the library about the issue.

  Kind regards,
  KIR

0
Comment actions Permalink

Hello Ray,

  Could you please try another experiment:

  Unpack TeamCity3/webapps/ROOT/update/plugins/svnAgent.zip
  Copy jsvn script near unpacked jars
  Try to run the checkout test again, with svnkit from your old TeamCity installation.

  Just to double-check that old svnkit works fast.

  Thanks,
  KIR

0
Comment actions Permalink

I can't be sure because I don't have my TC3 install any more (disk crashed, no backup ... I know, I know :-P), but I'm pretty sure we were using 3.1.1.

Some additional testing I was doing to see if it was the ssh handling of SVNKit or what ... I used jsvn from the command line to download Apache Geronimo source, which uses HTTPS; jsvn dowloaded all of Geronimo trunk, around 71000 files, in about 1 1/2 minutes. Regular svn took about the same.

0
Comment actions Permalink

Hello Ray,

  I got feedback from SVNKit developers that this is SVNKit problem.
  I've put their response (with some workaround) to TW-7412 .
  Hopefully, the problem will be fixed in the next TeamCity 4.1 EAP build (to be release in the beginning of the next week).

  Kind regards,
  KIR

0
Comment actions Permalink

Thanks for your help. I applied the patched trilead jar and things seem back to normal. All is well.

0

Please sign in to leave a comment.