Subversion stability in TC 4.0.2

Hi!

I am having issues with one of our subversion servers at work. This should not really be more than an annoyance as TeamCity usually handles this nicely and shows that svn-connection status on the overview-page.

Unfortunately TC will sometimes fail the actual builds when there is a subversion connection problem.

Is there any way that TC could avoid reporting these problems as build failures and just keep trying to sync until it works instead (or perhaps only notify the system administrator of the problem and not all developers who monitor the build results)

Here are a few callstacks from different failures:

jetbrains.buildServer.serverSide.ExecutionException: Unable to collect changes
jetbrains.buildServer.serverSide.ExecutionException: Unable to collect changes
at jetbrains.buildServer.serverSide.impl.BuildStarter$1.run(BuildStarter.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: org.tmatesoft.svn.core.SVNException: svn: connection refused by the server
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:205)
at java.util.concurrent.FutureTask.get(FutureTask.java:80)
at jetbrains.buildServer.serverSide.impl.CancelableTaskHolder.waitForTaskToComplete(CancelableTaskHolder.java:9)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.doCollectChanges(TopBuildDependencyGraphImpl.java:3)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.access$000(TopBuildDependencyGraphImpl.java:49)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:2)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$3.run(TopBuildDependencyGraphImpl.java:1)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl$4.run(TopBuildDependencyGraphImpl.java:1)
at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:24)
at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:31)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.runAsSystem(TopBuildDependencyGraphImpl.java:60)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChanges(TopBuildDependencyGraphImpl.java:54)
at jetbrains.buildServer.serverSide.impl.dependency.TopBuildDependencyGraphImpl.collectChanges(TopBuildDependencyGraphImpl.java:75)
at jetbrains.buildServer.serverSide.impl.BuildServerImpl$5.run(BuildServerImpl.java:4)
... 6 more
Caused by: jetbrains.buildServer.vcs.VcsException: org.tmatesoft.svn.core.SVNException: svn: connection refused by the server
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.getVersionDisplayName(SvnSupport.java:165)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.createRepositoryVersion(VcsChangesLoader.java:149)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChanges(VcsChangesLoader.java:128)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:667)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:72)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:2)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1)
... 5 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:82)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:62)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:73)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1234)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:95)
at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:279)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.getVersionDisplayName(SvnSupport.java:159)
... 11 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:53)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:53)
... 16 more



and another one

jetbrains.buildServer.vcs.VcsException: org.tmatesoft.svn.core.SVNException: svn: Error during update from svn://ganon/Projects/Core/Database/trunk/assets to C:\TeamCity\buildAgent\work\19dc7f0f4800369e\assets\core
svn: connection refused by the server
jetbrains.buildServer.vcs.VcsException: org.tmatesoft.svn.core.SVNException: svn: Error during update from svn://ganon/Projects/Core/Database/trunk/assets to C:\TeamCity\buildAgent\work\19dc7f0f4800369e\assets\core
svn: connection refused by the server
at jetbrains.buildServer.vcs.svn.SvnCheckoutOnAgentSupport.updateSources(SvnCheckoutOnAgentSupport.java:104)
at jetbrains.buildServer.agent.impl.patch.GetProjectSources.checkoutSources(GetProjectSources.java:278)
at jetbrains.buildServer.agent.impl.patch.GetProjectSources.execute(GetProjectSources.java:100)
at jetbrains.buildServer.agent.impl.runStages.GetSourcesStage.doBuildStage(GetSourcesStage.java:31)
at jetbrains.buildServer.agent.impl.BuildRunAction.doStages(BuildRunAction.java:115)
at jetbrains.buildServer.agent.impl.BuildRunAction.run(BuildRunAction.java:81)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.tmatesoft.svn.core.SVNException: svn: Error during update from svn://ganon/Projects/Core/Database/trunk/assets to C:\TeamCity\buildAgent\work\19dc7f0f4800369e\assets\core
svn: connection refused by the server
at jetbrains.buildServer.vcs.svn.TCSvnUpdateClient.doCheckout(TCSvnUpdateClient.java:77)
at jetbrains.buildServer.vcs.svn.SvnCheckoutOnAgentSupport.updateSources(SvnCheckoutOnAgentSupport.java:73)
... 6 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: connection refused by the server
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:82)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:62)
at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:73)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1234)
at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.getLatestRevision(SVNRepositoryImpl.java:168)
at org.tmatesoft.svn.core.wc.SVNBasicClient.getRevisionNumber(SVNBasicClient.java:479)
at org.tmatesoft.svn.core.wc.SVNBasicClient.getLocations(SVNBasicClient.java:848)
at org.tmatesoft.svn.core.wc.SVNBasicClient.createRepository(SVNBasicClient.java:531)
at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:830)
at jetbrains.buildServer.vcs.svn.TCSvnUpdateClient.doCheckout(TCSvnUpdateClient.java:74)
... 7 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:53)
at org.tmatesoft.svn.core.internal.io.svn.SVNPlainConnector.open(SVNPlainConnector.java:53)
... 15 more


and another one

jetbrains.buildServer.vcs.VcsException: java.lang.RuntimeException: svn: connection refused by the server
jetbrains.buildServer.vcs.VcsException: java.lang.RuntimeException: svn: connection refused by the server
at jetbrains.buildServer.vcs.svn.SvnCheckoutOnAgentSupport.updateSources(SvnCheckoutOnAgentSupport.java:112)
at jetbrains.buildServer.agent.impl.patch.GetProjectSources.checkoutSources(GetProjectSources.java:278)
at jetbrains.buildServer.agent.impl.patch.GetProjectSources.execute(GetProjectSources.java:100)
at jetbrains.buildServer.agent.impl.runStages.GetSourcesStage.doBuildStage(GetSourcesStage.java:31)
at jetbrains.buildServer.agent.impl.BuildRunAction.doStages(BuildRunAction.java:115)
at jetbrains.buildServer.agent.impl.BuildRunAction.run(BuildRunAction.java:81)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.RuntimeException: svn: connection refused by the server
at jetbrains.buildServer.vcs.svn.TCSvnUpdateClient.getRevision(TCSvnUpdateClient.java:55)
at jetbrains.buildServer.vcs.svn.TCSvnUpdateClient.doCheckout(TCSvnUpdateClient.java:74)
at jetbrains.buildServer.vcs.svn.SvnCheckoutOnAgentSupport.updateSources(SvnCheckoutOnAgentSupport.java:73)
... 6 more

5 comments
Comment actions Permalink

Hello Thomas,

  The exceptions you posted indicate subversion access problem, which occured after collecting the changes,
  when build was already started. I'm afraid, when problem occurs on this stage, TeamCity has to report a error for
  the build, because we cannot distinguish temporary problem from permanent one.

  We could probably cancel such builds and re-add them to the build queue, but this may lead to endless build loop.
  What do you think?

  Another question is why do you have such connection failures to svn server, what is the primary cause of the problem?

  Regards,
  KIR

0
Comment actions Permalink

If the IT department can solve the issue then that will eliminate the need for a fix in TC for us.

I just thought that the appropriate way for TC to handle this would be to show the ordinary red bar that TC usually shows on VCS connection problems.

BTW, while I'm at it. Is there any way to control the timeout used by TC when trying to connect to the subversion server?

0
Comment actions Permalink

Hello Thomas,

  You can specify only read timeout for http/https connections (and only in TeamCity 4.1 EAP).
  For svn:// repositories, there is no timeout (according to the code).

  And error like "connection refused" means that connection is not simply accepted.

  You may also take a look at the TeamCity polling interval setting: http://url.ie/1e9p


  Kind regards,
  KIR

0
Comment actions Permalink

Hello Kirill,

Thank you for your quick responses.

We will of course have to resolve the IT-issues here in any case (the repos are located in different offices and accessed over vpn). Seems like the build server is stressing the svn:// connections more than usually happens and this must have exposed some flaw we have always had. Seems like we might have more stable connections via file:// so I hope that will solve our issues.

If you can figure out a way to handle bad connections more gracefully it would be great though. I believe it would be a fine start to just be able to filter the notifications so that developers can choose not be notified when the build fails because of svn issues. I want to avoid the situation where the builds fail so often on bogus errors that people start ignoring the build server.

Kind regards,
Thomas

0
Comment actions Permalink

Hello Thomas,

  If subversion server load is caused by TeamCity, please try increasing VCS poll interval.
  Please note, that each VCS root in TeamCity polls your subversion server each 2 minutes by default,
  even when builds are not running. If svn stability is caused by TeamCity load, consider increasing poll interval,
  to, say, 5 minutes.

  Regards,
  KIR

0

Please sign in to leave a comment.