Build always hangs on Checking for changes - SVN

TeamCity 5.1.2

I've been trying to evaluate for a couple of weeks already, but I can't even get the thing to checkout sources from the repository. All not looking good. Please don't disappoint me with this product.

svn:// protocol

Checking out just one root, no externals. Selected ignore externals.

Testing connection works fine.

Tried running build both on the server and on the agent - no difference.

As soon as I start the build it goes into "checking for changes" forever. Clicking on the "thread dump" opens a little window and hangs forever and does not display thread dump.

Checkout rule is simple +:.=>mysubdirectory

Checkout directory is X:\mydirectory


Other notes:

1. Sometimes after running a build for several hours I do get 2-3 changes listed as "pending for the next build" in the "current status".

2. Just _once_ among my attempts I had the build actually checking out code - e.g. I saw the directory created on the disk. But then it failed for some other reason. After that I got back to the usual "checking for changes" forever.

Please help.

14 comments

Hello,

   The problem is indeed strange.

   Checking for changes operation works on the server-side only, so checkout mode or checkout directory doesn't matter in your case.
   It is really strange that it executes so long, given that externals are turned off.

   What is the rough size of your repository? How long does "svn log -l 100" command executes on it?

   Could you please take a thread dump of the server from the interface http://your.server/admin/serverConfig.html?init=1&tab=diagnostic and send it to us?

  Could you also enable debug logging on the same page, run the build, and send us log archive from http://your.server/admin/serverConfig.html?init=1&tab=serverLogsViewer page?
  Don't forget to disable debug logs after that, they slow down the server!

  Sorry for the problem,

  Best regards,
  KIR

0

The repository is not huge as I understand. The total codebase is about 500K lines of code, I am only trying to checkout just one portion. Connection is a 100Mbps local network. There is no load on the SVN server.

svn log -l 100 takes about 1 second to execute.

Thread dump using the admin interface - attached.



Attachment(s):
threadDump-2010-08-04_07.32.37.txt.zip
0

Cannot send the debug log yet because I got the build running a few more times, then hanging again and now it is running.
I changed VCS root setup from Ignore Externals to Full Externals and the build went on. Then I added another VCS root (it was also in Ignore Externals) and that caused the build to hang again. Then I switched the new root to Full Externals but the build still hang. Restarting TeamCity caused the build to run again.
I will upgrade to 5.1.3 and run a few more times if I come across the problem, will post the debug logs.

0

Hello,

  According to the thread dump, TeamCity awaits response from your Subversion server.
  I'm unsure why does it take so long. May be this is a load of the SVN server or something like this.

  "Ignore externals" mode should work faster than with "Full externals".

  Will away for your feedback after 5.1.3 is installed.

  Regards,
  KIR

0

Hi Kirill,

I tried 5.1.3 and it worked well once. Then it hang on checking for changes once. Then I cleaned the checkout directory myself and 5.1.3 worked once again. Now it went back to checking for changes always. When it works, it changes to "Running" almost instantly.

Well, I tend to think there is some problem with the SVN client that TeamCity uses or a problem in the communication between TeamCity and the SVN client.

There is no load on the SVN server and we had no problems with our SVN server. We use TortoiseSVN client and never had it hanging like this - all works very fast. TeamCity is not slow - it just does not talk to the SVN server properly.

The server that we use: svnserve.exe 1.6.12.38263 for Windows. The exe is 32bit. I downloaded and installed VisualSVN Server 2.2.13, but we disable the VisualSVN Server and register svnserve.exe found in VisualSVN Server's bin directory to run as a windows service.

Restarting svnserve.exe helped TeamCity now. BTW TortosiveSVN was still working fine when TeamCity was hanging. Will see how many times it can run before hanging again. Also, I will try a different svnserve.exe.

0

The build runs a few times, then goes to "checking for changes". I tried restarting the agent service and the TeamCity web server instead of restarting svnserve and that also helped to get it building again (not sure whether restarting the agent or the web server helped). So at least I can continue playing with the thing - just restarting it.

Now here is another problem.

This build is probably hanging. Last message was received on: 07 Aug 10 00:24 (16m:36s ago)


Started:

Build is running
07 Aug 10 00:12   Cannot stop
Agent: awdev7
Time left: overtime 22m:39s (passed: 23m:56s, estimate: 1m:17s) Triggered: by you on 07 Aug 10 00:11
Thread dump: View thread dump
Running step: Updating sources: Agent side checkout... : VCS Root: XXX: Checkout on agent does not support interrupting. Waiting for the operation to fi

Now I have a build that I tried to Cancel, but even that has failed. Restarting the agent service or the web server does not help. Getting thread dump does not work. As far as I can tell the server does nothing apart from periodic refreshes of the webpage. In the meantime TortoiseSVN clients can access my SVN server no problems. I think TeamCity locks itself in many ways. Maybe a bit overengineered?

1. How can I clear this?

2. Any chance to resolve "checking for changes" hanging?

Thanks

Roman

0

Hi Kirill,

Here are some good logs for you.

First build run works well. Second build hangs on checking for changes. I stopped it. Third build hangs on checking for changes too. I am taking a thread dump here. Stopped the build. Downloaded the server logs.

Now if I restart TeamCity service, then the build will work for one or several times okay. The hanging is intermittent, but always reproducible. Accessing the SVN server via TortoiseSVN is not a problem during any of this.



Attachment(s):
threadDump-2010-08-07_19.35.22.txt.zip
teamcity_server_logs_20100807.zip
0

Hello Roman,

  TeamCity uses SVNKit library to access SVN repositories.
  May I ask you to try downloading the latest version of the library (from here). After that, please unpack it and try running command
  "jsvn.bat ls <your repository>" several times, to see how it works for you. "jsvn.bat" accepts similar syntax as command line svn client.

  If this works for you, please replace svnkit.jar library in your TeamCity installation with the one from downloaded SVNKit (backup old one first).
  The location of svnkit.jar on the server is TeamCity/webapps/ROOT/WEB-INF/plugins/svn/server.

  If this helps with Changes collecting problem, try switching TeamCity to "checkout on server", because in this case TeamCity will use new
  SVNKit library to checkout sources as well.

  I'm sorry for the problem you're experiencing. I'll try to call SVNKit developers to look at the problem.

  Best regards,
  KIR

0

Hi Kirill,

jsvn.bat works well. No hanging observed. I run jsvn checkout a number of times and jsvn ls too. This was on the same machine as teamcity.

I have copied jsvn jars to teamcity location that you specified (I copied all jars that I thought applicable, not just one). I have also found some teamcity jars what I believe to be for the agent and copied the new jsvn jars over them too.

I tried checkout on the server and checkout on the agent - no change. After a few (1-3) successful runs - it does into the dreadful "checking for changes" state.

0

Hello,

   I think the hang may be somehow connected to the simultaneous multiple connections to the SVN server. Are you sure that you've setup svnserve.exe correctly and that svnserve.exe from the VisualSVN Server distribution may be installed separately as you did it? There may be a chance that your installation doesn't support multiple simultaneous connections in a row, or something like this.

  I'm asking because I'm a little bit confused - from the thread dump I see several threads which are awaiting for data from the connections to SVN server, and these connections hang (no incoming data). So there is either something on the SVN server side (some lock or anyting) or incorrect handling of incoming data on the TeamCity/SVNKit side - but there is nothing going on there, no deadlocks, infinite cycles, nothing.

  I searched for a way to debug svnserve.exe, but found only these instructions.  A traffic snapshot between TeamCity and your server may help, at least I'd try to get it to obtain additional information.

  Sorry, couldn't suggest anything else so far.

  Best regards,
  KIR

0

I changed from using svnserve.exe supplied with VisualSVN to svnserve.exe supplied with CollabNet 64-bit distribution - did not make any difference.

The last thing I am going to try is running TeamCity on real hardware that is planned to become a build server, rathern than on a VM for testing.

0

Hello,

   Another idea - do you probably use any antivirus software on your TeamCity server installation? If so, could you try disabling it? We've had some experiences with odd behaviour server when antiviruses were installed. Just a thought.

  Also, we've added a read timout for SVN connections, and this fix will be available in 5.1.4 (which will be probably released on Monday or earlier).

  Best regards,
  KIR

0

Kirill,

I have been experiencing similar problems with SVN.  I posted a thread here and at that time we were experiencing more issues with CVS than with SVN:
http://devnet.jetbrains.net/message/5269672
We are moving to all SVN in the next week or so I didn't push the CVS issue that much.

What I was seeing was that builds would run fine for some period of time and then eventually the would just start hanging on the Checking for Changes step.  I had spent time diagnosing our network and SVN server and could not find any problems.
I had given up because we were planing on moving to new hardware for our TC server ( from an old Windows box to a new 24-core Linux server ).  We just completed the move and after several days of good performance, I am seeing a few builds hanging again.
Currently I have one that has been running overtime 31 minutes ( normally total build time is 15 minutes ).  At the same time other builds are running using the same VCS roots and are not hanging.

I get this message trying to get the thread dump:
TeamCity was unable to locate any processes of this build.

I have also just recently moved to 6.0 ( 15311 ) from 5.1 and we experienced the same problem on both versions.

I have also noticed many of these errors in the system VCS logs ( I did not see this error in 5.1 )

SQL query: INSERT INTO vcs_history (MODIFICATION_ID, USER_NAME, DESCRIPTION, CHANGE_DATE, VCS_ROOT_ID, VCS_ROOT_VERSION, VERSION, DISPLAY_VERSION, CHANGES_COUNT, REGISTER_DATE) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
SQL exception: Incorrect string value: '\xEF\xBF\xBDANS...' for column 'description' at row 1

We are due to move into production with several builds very soon.  I am worried about the stability of the environment running in TC.  
Our builds that are running in TC are also currently running stand-alone in parrallel and the stand alone builds do not experience issues with our source control systems.


thanks for your help.
NSN

0

Hello,

   Regarding the thread dump for diagnose SVN checking for changes hangs: to take a thread dump, please go to Server Configuration page of Administration area, tab Diagnostics.
   There is a control which allows to take the thread dump of the server (and this thread dump should indicate where the hang occurs).

   Regarding SQL issue in the log - which database do you use? How did you specify the encoding for database tables and for database connection?

   Also, it would be helpful if you attach your teamcity-vcs logs for closer investigation.

  Thanks,
  KIR

0

Please sign in to leave a comment.