Build hangs checking for changes

Hi all,

I have a build that has consistently started hanging during the 'checking for changes' phase. The final build log looks something like this:

[21:13:28]: Checking for changes
[21:24:07]: Clearing temporary directory: C:\BuildAgent\temp\buildTmp
[21:24:07]: Checkout directory: C:\BuildAgent\work\1bfaefc487599a54
[21:24:07]: Updating sources (4h:01m:26s)
[21:24:07]: [Updating sources] Updating project sources (4h:01m:26s)
[21:24:07]: [Updating project sources] dh trunk; revision: 938_2009/06/05 21:24:07 +0100
[01:25:30]: [Updating project sources] Subversion update_external problem for C:\BuildAgent\work\1bfaefc487599a54\BuildScripts: svn: timed out waiting for server
[01:25:34]: [Updating sources] jetbrains.buildServer.vcs.VcsException: Subversion: checkout failed
jetbrains.buildServer.vcs.VcsException: Subversion: checkout failed
at jetbrains.buildServer.vcs.svn.SvnCheckoutOnAgentSupport.updateSources(SvnCheckoutOnAgentSupport.java:92)
at jetbrains.buildServer.agent.impl.vcs.CheckoutByIncludeRuleAdapter$1$1.process(CheckoutByIncludeRuleAdapter.java:28)
at jetbrains.buildServer.agent.impl.vcs.CheckoutByIncludeRuleAdapter$2.updateSources(CheckoutByIncludeRuleAdapter.java:49)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent.checkoutSources(ProjectSourcesOnAgent.java:61)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.checkoutSources(ProjectSourcesGetter.java:269)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.execute(ProjectSourcesGetter.java:123)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doGetSources(CheckoutSourcesStage.java:45)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doRecoverableStage(CheckoutSourcesStage.java:31)
at jetbrains.buildServer.agent.impl.runStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:61)
at jetbrains.buildServer.agent.impl.BuildRunAction.doStages(BuildRunAction.java:135)
at jetbrains.buildServer.agent.impl.BuildRunAction.access$000(BuildRunAction.java:21)
at jetbrains.buildServer.agent.impl.BuildRunAction$1.run(BuildRunAction.java:91)
at java.lang.Thread.run(Thread.java:595)

[01:25:34]: Build finished


Oddly, I have other projects in TeamCity which share the same VCS root which are building just fine...


Debugging steps I've tried:

  • I've checked I can manually update the working directory using TortoiseSVN and that seems fine
  • Started and stopped the build in question. The problem is repeateable.
  • I've also copied the build configuration to see if the copy also exhibits the problem and it does
  • Upgraded to TeamCity 4.5.1 (problem started while on 4.5)
  • Switched on full debugging in TeamCity (nothing jumps out at me from the logs)


Any help or suggestions for debugging would be gratefully receieved. I'm in the process of introducing our team to Teamcity. Initial feedback has been excellent so I don't want anything to rock the boat!

Thanks,


Andy

5 comments
Comment actions Permalink

Hello Andy,

  Do I understand correctly that the same VCS root was used successfully for other build configurations?
  What is the difference between them? Do you use checkout rules?

  Could you probably try checkout on server mode instead of checkout on agent? Would the result differ?

  What are you settings for Subversion root, what protocol do you use to access repository?


  Please make the following test: download org.tmatesoft.svn_1.2.3.5729.standalone.zip file, unpack it,
  and run jsvn checkout <your repo> command. This is SVNKit library used in TC 4.5.1.
  Will it work?

  In fact, we're going to release TeamCity 4.5.2 today or tomorrow, and it will contain updated subversion library (1.3.0).

  Kind regards,
  KIR

0
Comment actions Permalink
Kirill,



Thanks for the quick response. Always appreciate the prompt support from you guys.



To answer your question, yes, the same VCS root is being used successfully for other build configurations. We are not using checkout rules. Each configuration just inkoves a different NAnt script which builds a certain project out of the shared root.



Our SVN settings are pretty standard. Protocol is https. The root has one svn:external.


jsvn checkout from the link you sent works fine for me checking out a clean working copy.




Interestingly, my copy of the original build configuration just ran:



[09:43:26]: Checking for changes
[11:20:43]: Clearing temporary directory: C:\BuildAgent\temp\buildTmp
[11:20:43]: Checkout directory: C:\BuildAgent\work\1bfaefc487599a54
[11:20:43]: Updating sources (16m:35s)
[11:20:43]: [Updating sources] Updating project sources (16m:35s)
[11:20:43]: [Updating project sources] AllProjects trunk; revision: 951_2009/06/08 11:20:43 +0100
[11:20:43]: [Updating project sources] Will use fast SVN update
[11:26:57]: [Updating project sources] Local changes will be reverted for C:\BuildAgent\work\1bfaefc487599a54\SomeProject\SomeProjectFolder\SomeProject.swf; status: conflicted; properties: changed
[11:27:02]: [Updating project sources] Local changes will be reverted for C:\BuildAgent\work\1bfaefc487599a54\SomeProject\Flex\SomeProject\src\com\mycomany\SomeProject\business\Services.mxml; status: conflicted; properties: unchanged
[11:28:24]: [Updating project sources] Subversion update_external problem for C:\BuildAgent\work\1bfaefc487599a54\BuildScripts: svn: 'C:\BuildAgent\work\1bfaefc487599a54\BuildScripts' is not a working copy
[11:28:47]: [Updating project sources] Clean checkout: Subversion update_external problem for C:\BuildAgent\work\1bfaefc487599a54\BuildScripts: svn: 'C:\BuildAgent\work\1bfaefc487599a54\BuildScripts' is not a working copy
[11:28:47]: [Updating project sources] Cleaning C:\BuildAgent\work\1bfaefc487599a54
[11:28:47]: [Updating project sources] Will use fast SVN update
[11:37:19]: Starting build process in C:\BuildAgent\work\1bfaefc487599a54

Which looks like it had some trouble with the svn:external. I've kicked off another build of the original configuration to see if works now. 50 minutes though, and still checking for changes.

I'll try a server-mode checkout and see if that helps. As we're still evaluating TeamCity it's still using its internal database. Could that ever cause a problem like this?

Thanks for the help,


Andy

0
Comment actions Permalink

Hello Andy,

   Using internal database is definitely not the issue here.

   What I could suggest is to disable checking for changes in externals, if it takes so much time - this can be done on the VCS root page. The drawback is that changes in externals won't be shown on the TeamCity UI.

   I'd also suggest trying to setup TeamCity profiling plugin from http://www.jetbrains.net/confluence/display/TW/Server+Profiling .

   I believe that there is a problem contacting external subversion repository. BTW, do your externals refer to the same repository, or to another one? Please check that externals syntax is correct and that externals repository is available.

  Kind regards,
  KIR

0
Comment actions Permalink

Kirill,

I'd written a long post describing my latest trouble-shooting findings then someone on another team here mentioned that they'd been experiencing some hangs with TortoiseSVN when updating their repository which also  links to the same svn:externals repository as ours. I restarted the Subversion server and now everything seems to have returned to normal on the TeamCity build server!

Apologies for the false alarm! Thanks for your help though. Much appreciated.


Andy

0
Comment actions Permalink

Hello Andy,

   Good news indeed. But if the problem will reappear again, please consider increasing the poll interval from TeamCity to VCS (may be, TeamCity creates too much load to your subversion server).
   You can increase poll interval individually for a checkout root, or change the default interval on the server setting page.

   Kind regards,
   KIR

0

Please sign in to leave a comment.