SVNServe processes/threads not being closed properly?

Hello Graham,

   How did you configure SVNServe to be started? I want to reproduce the issue so I'd like to mirror your configuration.

  Regards,
  KIR

I've started it via a script, directly from a sh terminal, and from an apple script all with the same results.

svnserve -d --listen-port 1983 -r ~/MyRepos/
and
svnserve -d --listen-port 1983 --threads -r ~/MyRepos/

these are running on the 1.6.x package I've installed from opencollab (http://subversion.apache.org/packages.html#osx). I've not tested with 1.4.4 which is bundled with OSX as our repos was previously using 1.5.5.
Also note the server is running leopard, not snow leopard.

When I get home tonight I'm going to have a look with strace, and I will post any significant (svn) .conf details too if you'd like. Are there any team-city logs you'd like? (I can give you admin access to our team city if you like...)

Hello Graham,

   I've tried your configuration with TeamCity 5.1.2 - so far, it works for me.
   I use

svn, version 1.6.6 (r40053)
   compiled Oct 22 2009, 14:13:09
Copyright (C) 2000-2009 CollabNet.

   An access to your TeamCity installation would be great. You can send the credentials to kirill.maximov@.

   Also, how do you check the number of svnserve processes on your machine?

   It is rather unlikely that TeamCity leaves open connections to svn, it uses common approach for different types of repositories and always closes the connection.

   Regards,
   KIR

I check the Activity Monitor in osx and after 10 mins or so (3 VCS roots on 8 configurations, checking for updates every 60 secs) I have 200+ "svnserve" processes (or about 190 threads in threaded mode) and from then on SVN reports (via teamcity) that it cannot open any more handles to the ~/Repos/Format file

Your email is kirill.maximov at JetBrains dot net? (the domain was omitted)

I only make this assumption that it might be team city leaving it open because I read somewhere (wrongly or rightly) that the connection stays open server side if it's not yet recieved an EOF (not sure if this is a double-terminator, or a token, again, I don't know the svn protocol) and I've had this server going (albiet on a different machine, see GP) for weeks on end with no trouble, I start using teamcity and it has these problems... But if it's a file/protocol/socket issue, I'm sure it could be easily down to an OSX/OS configuration difference...

OK, I see than, thanks. Will try to hunt the problem in place.
My email ends with jetbrains.com.

KIR

Also, there is a possible workaround to run SVNserve via LaunchDaemon.
I attach a plist file which runs svnserve on my machine, with a couple of tweaks you can use it on yours.
This way, you'll have svnserve daemon running as needed.

This file should be put to /Library/LaunchDaemons directory and started as
sudo launchctl load /Library/LaunchDaemons/org.tigris.subversion.svnserve.plist

May be you'll find this approach interesting. Not sure, whether it would help, though.

KIR

Hi Graham,

   Thanks for the access.

   The situation is rather interesting, because it is not reproducible on my side yet.

   Could you please send exact version of the SVN server you're using. If this is not the latest SVN server, please upgrade and see if the problem reproduces - may be this is not TeamCity's fault.

  Thanks!
  KIR

My mistake, svnserve reboots fine. I might try the LaunchD but atm I can control it quite nicely from the dock (I've got an apple script to start and stop it).

Anyway, version

svnserve, version 1.6.11 (r934486)
   compiled Apr 20 2010, 17:01:38

And now it's gone down. Teamcity's error is "svn: Handshake failed, received: '' "

The last entries in the log are;

2437 2010-06-16T18:55:03.570974Z 86.16.22.183 Teamcity Rocket open 2 cap=(edit-pipeline svndiff1 absent-entries depth mergeinfo log-revprops) / - -
1744 2010-06-16T18:55:08.522494Z 86.16.22.183 - - ERR subversion/svnserve/main.c 827 35 apr_proc_fork: Resource temporarily unavailable
2438 2010-06-16T18:56:03.555590Z 86.16.22.183 Teamcity TestGame open 2 cap=(edit-pipeline svndiff1 absent-entries depth mergeinfo log-revprops) / - -
2438 2010-06-16T18:56:03.557324Z 86.16.22.183 Teamcity TestGame get-latest-rev
1744 2010-06-16T18:56:03.562015Z 86.16.22.183 - - ERR subversion/svnserve/main.c 827 35 apr_proc_fork: Resource temporarily unavailable
1744 2010-06-16T18:56:08.548931Z 86.16.22.183 - - ERR subversion/svnserve/main.c 827 35 apr_proc_fork: Resource temporarily unavailable

Could not open a new pseudo-tty.

Hello Graham,

   I've installed the same version of SVN locally (but on the Snow Leopard). So far, I cannot reproduce the problem. The processes are started (I see this in logs), but finished quickly.

  On the server, I see you have some specific SVN configuration directories, for each VCS root. Why do you use them? Do you use any non-standard options for starting the server?

  Regards,
  KIR

Nothing non-standard other than the port. You can see the command line above.

I have individual repositories running from one svnserve (one repos root). This seems to run fine, I'm pretty sure it's a standard way of setting things up (been running happily for 2 years)

I simply use this line to create a new repos;

svnadmin create ~/TootleRepos/$repos

This is the "Sample" output from Activity Monitor in OSX of the first PID (presumably the main process)

Sampling process 6027 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling svnserve (pid 6027) every 1 millisecond
Call graph:
    2504 Thread_2507
      2504 start
        2504 main
          2504 accept
            2504 accept

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        accept        2504
Sample analysis of process 6027 written to file /dev/stdout

Sampling process 6079 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling svnserve (pid 6079) every 1 millisecond
Call graph:
    2266 Thread_2507
      2266 start
        2266 main
          2266 serve
            2266 svn_ra_svn_handle_commands2
              2266 svn_ra_svn_read_tuple
                2266 svn_ra_svn_read_item
                  2266 readbuf_getchar_skip_whitespace
                    2266 readbuf_getchar
                      2266 readbuf_fill
                        2266 readbuf_input
                          2266 svn_ra_svn__stream_read
                            2266 svn_stream_read
                              2266 sock_read_cb
                                2266 read
                                  2266 read

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        read        2266
Sample analysis of process 6079 written to file /dev/stdout

Hello Graham,

  I suppose your system has some kind of proxy or some kind of middle man like that. Or there is some specific processing of the svn port you're using.

  I think so, because I've changed the hostname in the svn path in your VCS roots to 'localhost', and after that the problem has disappeared completely.

  But, there are still ESTABLISHED svnserve connections to some address ....know.cable.virginmedia.com, which are not terminated. These connections have apeared when I run checking for changes with old svn address.

  In your setup I created 2 build configurations, one runs netstat, another runs lsof, just to see what's going on.

  I believe the problem is not related to TeamCity, otherwise we'd see the same hangs for localhost address as well, I think.

  Best regards,
  KIR

I hadn't thought of that at all! There are 2 middle men (maybe 3 technically including the modem).

I use dyndns to forward XXXX.getmyip.com to my router, and the router is port-forwarding (1983 to fry.local). XXX.know.cable.virginmedia.co is my hostname on my internet connection (ie. the modem's hostname)

As you can see I normally use xxxx.Getmyip.com internally (even when the machines are on the same LAN or even same machine) just to make sure the external domain is working for others,
I do this with svnx, tortoiseSvn etc with no problem...

Thanks for tracking all this down, I'll use localhost for now and do a bit more investigation as to which part is causing this problem, as I believe there is still a problem, (again, other svn clients don't have a problem using the external hostname to access) but where blame lies is maybe a little unclear now. :)
I wouldn't want the problem to come up again if I move the svn server to another machine, or move team city...

I'll check the localhost name is fully working when I get home tonight/tomorrow

Thanks again