TeamCity against Subversion with SSPI

We're running TeamCity with about 15-20 projects all hitting a SVN repository that's hosted on Windows. The authentication is done via the SSPI module in apache. We're having some performance problems with the SVN server, its getting constantly pegged and is blocking out our developers. We suspect the problem is caused by SSPI, though we're uncertain. Does anyone have experience using SSPI on their SVN server with TeamCity? Any similar problems?


Specifics:
TeamCity 3.1
Apache 2.0.61
SSPI 1.0.4
SSL 0.98g

10 comments

Thomas,

You may consider increasing "VCS changes check interval" that is how frequently TeamCity checks for changes in each VCS root.
The default setting can be changed on the Server Settings and each VCS root can override the setting also.

How many VCS roots do you have? Do you use externals?

TeamCity should not open more then 5 connections to the VCS servers to check for changes. Each running build can also add one additional connection.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Hello Thomas,

This is the answers from Alexander Kitaev, an SVNKit developer.

Does your developer know if SSPI is detrimental to a SVN instance?


In general SSPI is an interface specification, thay may be implemented
by different security providers. SVNKit - Subversion library used by
TeamCity - only supports NTLM authentication protocol and this support
is limited, e.g. it may not work properly with all NTLM-enabled servers.

When NTLM authentication method is used, it may result in significant
connection overhead, as several HTTP requests are needed to authenticate
user and authentication may be performed several times during the same
session.

Most of NTLM-enabled servers, however, provides Basic authentication
method as a fallback that is preferable from performance point of view.
To make SVNKit use Basic authentication whenever it is available, one
may set system property (see TeamCity documentation on how to set system
property for build agent or server):
svnkit.http.methods=Basic,Digest,NTLM - setting such property will
override default order available authentication methods where NTLM is
usually the first one.

Do you guys have any sort of recommended hardware configurations for
the SVN server?


I'd say that there is no specific recommendations on hardware
configuration for Subversion, the disk IO and processor speed are
probably more important than amount of memory. However, in case Apache
server is used, it is important to understand that each user's request
is processed within a separate thread or process, so ability to run
multiple thread simultaneously (multi-core processor or several
processors are better than single-core one) and corresponding apache
configuration setting (that may set certain limit on amount of processes
or threads to process requests) are important.

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java Versioning Library!

We're running TeamCity with about 15-20 projects all hitting a SVN
repository that's hosted on Windows. The authentication is done via
the SSPI module in apache. We're having some performance problems with
the SVN server, its getting constantly pegged and is blocking out our
developers. We suspect the problem is caused by SSPI, though we're
uncertain. Does anyone have experience using SSPI on their SVN server
with TeamCity? Any similar problems?

Specifics:
TeamCity 3.1
Apache 2.0.61
SSPI 1.0.4
SSL 0.98g

--
Sergey Anchipolevsky
Software Developer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


0

We have several projects also and have been noticing SVN access to be slowing down. I noticed my apache log file was 22Gb and 99% caused by the "buildagent" polling for changes.

I agree with the suggestion to increase the polling interval. But in looking at the log file it appears each VCS checkout is causing more traffic. I set the interval for 120seconds yet there are 20 hits every second to the Apache server.

It should be possible to check the root of the repository for a change every 120 seconds. If there are changes, then check each VCS checkout paths to figure out the specific change. It would greatly reduce the impact of TeamCity on Subversion.

btw, we also use SSPI for authentication but created a special non-Windows user account for TeamCity so we don't notice that part of the issue. but prehaps hitting the domain 20times a second to authenicate causes issues.

-Steve

0

Steve, how do you configure apache to allow the multiple auth? I think this may be the best way to solve our problem, but I haven't been able to set it up on our test server. When I use svn.exe with the basic auth credentials, it won't let me authenticate and I see the domain name in the apache error log.

0

Steve,

I agree with the suggestion to increase the polling interval. But in looking at the log file it appears each VCS checkout is causing more traffic. I set the interval for 120seconds yet there are 20 hits every second to the Apache server.


This can happen if a single check takes more then 120 seconds or close to that.
The load is obviously increased with the VCS roots number increase. Also, SVN roots with externals enabled require much more server communication (an additional request is performed for each directory under the root).

How many roots do you have and how many of them have externals enabled?

You can also see the TeamCity VCS-related activity in the logs/teamcity-vcs.log server log.

It should be possible to check the root of the repository for a change every 120 seconds. If there are changes, then check each VCS checkout paths to figure out the specific change. It would greatly reduce the impact of TeamCity on Subversion.


TeamCity should work exactly this way, provided externals are disabled in the VCS root settings and the single VCS root is used in all the build configurations , limiting the necessary sources subtree via checkout rules.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

To have two different authentication methods, you need to different ]]> entries in your Apache httpd.conf file. You would then unfortunately would have two different URLs for the same respository directory. (ex. "http://apache_server/svn/trunk/project1" and "http://apache_server/svn_for_buildagent/trunk/project1"). You will run into problems with this solution if you use "svn:externals" since they are usually "absolute URLs". This means the externals will either silently fail or error out with TeamCity when authenticating using the buildagent username. But, just today a new Subversion 1.5 was released that supports relative URLs for "svn:externals" to get around this problem.

0

I have about 50 VCS roots which I admit needs to be cleaned up a bit.

I think out of that only a few have externals turned on.

I don't think a single VCS root will work for use for two reasons. It would be a huge checkout and we have build agents in different offices so it would take forever. And we need fine grain check out since that is used for the triggering of the builds. We don't want every commit to trigger every TeamCity project to rebuild.

With subversion, it should be possible to check if anything changed on the entire server with one "log" on the root. That way it is possible to avoid checking all 50 VCS roots if edits occured. It is not guaranteed that all the VCS roots are on the same subversion server but that is our situation and probably most situations.

0

Steve,

I think out of that only a few have externals turned on.


I guess you can try to find the roots that are mostly causing the load by analyzing teamcity-vcs.log and SVN server access logs.

I don't think a single VCS root will work for use for two reasons. It would be a huge checkout and we have build agents in different offices so it would take forever.
And we need fine grain check out since that is used for the triggering of the builds. We don't want every commit to trigger every TeamCity project to rebuild.


Can you try to use a single VCS root and corresponding checkout rules for the specific configurations? A checkout rule limits the sources used for a build only to the directories specified. The build is triggered only if the mapped sources are changed.

Also, to minimize externals overhead, you may consider disabling the externals in the global root and creating more specific roots with externals enabled.


--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Thanks for clarifying the checkout rules. I didn't realize they allow removing paths and those would be excluded from the triggering of builds. My current situation is not severe enough the redo all of our checkouts but it nice to know there is way to minimize traffic.

One ability of TeamCity that is a big improvement over rivals (specifically CruiseControl.NET) is the reusing of checkouts. It saves a lot of configuring. But I always have to use checkout rules to move the directories into the correct relative location (I wish there were default checkout rules). You solution means more configuring per checkout so I wait until the problem is severe enough to warrent the effort.

I will turn off all svn externals for the few I mentioned since I actually am not using them.

0

Steve,

I've filed a request on ability to define default checkout rules for a VCS root: TW-5337. You can vote/watch for it.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Please sign in to leave a comment.