Subversion connection errors
Hi,
We have TeamCity<-->SVN problems. We get the following errors (logs are pasted below):
- org.tmatesoft.svn.core.SVNException: svn: Unknown error
- 500 Internal Server Error (https://svn)
- ClientAbortException: java.net.SocketException: Broken pipe
The errors occur "randomly". The erros occur on a number of different builds.
Anyone that have seen these kind of SVN errors? Any idea what the root cause is? Any idea how to solve them?
We are using TeamCity Enterprise Version 4.5 (build 8944), SVN version 1.5.6
Case 1:
[17:48:53]: Changes are already collected
[17:48:53]: Clearing temporary directory: C:\ba\temp\buildTmp
[17:48:53]: Checkout directory: C:\ba\wk\dd685567fbd8e7ae
[17:48:53]: Updating sources (2m:38s)
[17:48:54]: [Updating sources] Will perform clean checkout
[17:48:54]: [Updating sources] Clean checkout reasons
[17:48:54]: [Clean checkout reasons] The project sources on the agent are newer than requested
[17:48:54]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[17:48:54]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[17:49:32]: [Updating sources] Building incremental patch over the cached patch
[17:49:35]: [Updating sources] Building clean patch for VCS root: https://svn/Tools/tags/2009-Q3_1
[17:51:32]: [Updating sources] Problem while loading patch data stream: chunked stream ended unexpectedly
[17:51:32]: Will repeat attempt when server will be available, number of attempts left: 2
[17:51:42]: Updating sources (2m:26s)
[17:51:42]: [Updating sources] Will perform clean checkout
[17:51:42]: [Updating sources] Clean checkout reasons
[17:51:42]: [Clean checkout reasons] The project sources on the agent are newer than requested
[17:51:42]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[17:51:42]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[17:52:08]: [Updating sources] Building incremental patch over the cached patch
[17:52:12]: [Updating sources] Building clean patch for VCS root: https://svn/Tools/tags/2009-Q3_1
[17:52:17]: [Updating sources] Failed to build patch for build #85, build id: 70787, VCS root: https://svn/Tools/tags/2009-Q3_1, due to error: org.tmatesoft.svn.core.SVNException: svn: Unknown error
[17:54:08]: [Updating sources] Problem while loading patch data stream: chunked stream ended unexpectedly
[17:54:09]: Will repeat attempt when server will be available, number of attempts left: 1
[17:54:19]: Updating sources (2m:30s)
[17:54:19]: [Updating sources] Will perform clean checkout
[17:54:19]: [Updating sources] Clean checkout reasons
[17:54:19]: [Clean checkout reasons] The project sources on the agent are newer than requested
[17:54:19]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[17:54:19]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[17:54:49]: [Updating sources] Building incremental patch over the cached patch
[17:54:52]: [Updating sources] Building clean patch for VCS root: https://svn/Tools/tags/2009-Q3_1
[17:54:53]: [Updating sources] Failed to build patch for build #85, build id: 70787, VCS root: https://svn/Tools/tags/2009-Q3_1, due to error: org.tmatesoft.svn.core.SVNException: svn: Unknown error
[17:56:49]: [Updating sources] Problem while loading patch data stream: chunked stream ended unexpectedly
[17:56:49]: [Updating sources] java.io.IOException: chunked stream ended unexpectedly
java.io.IOException: chunked stream ended unexpectedly
at org.apache.commons.httpclient.ChunkedInputStream.getChunkSizeFromInputStream(ChunkedInputStream.java:252)
at org.apache.commons.httpclient.ChunkedInputStream.nextChunk(ChunkedInputStream.java:221)
at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:176)
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)
at jetbrains.buildServer.util.TCStreamUtil.writeBinary(TCStreamUtil.java:92)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.copyStreamToFileAndCloseStream(ProjectSourcesGetter.java:332)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.copyPatchToTempFile(ProjectSourcesGetter.java:443)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.loadAndApplyPatch(ProjectSourcesGetter.java:213)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.access$400(ProjectSourcesGetter.java:31)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter$4.execute(ProjectSourcesGetter.java:282)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.executePatchProcess(ProjectSourcesGetter.java:159)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.applyPatch(ProjectSourcesGetter.java:280)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.execute(ProjectSourcesGetter.java:119)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doGetSources(CheckoutSourcesStage.java:45)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doLastAttempt(CheckoutSourcesStage.java:37)
at jetbrains.buildServer.agent.impl.runStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:57)
at jetbrains.buildServer.agent.impl.BuildRunAction.doStages(BuildRunAction.java:135)
at jetbrains.buildServer.agent.impl.BuildRunAction.access$000(BuildRunAction.java:21)
at jetbrains.buildServer.agent.impl.BuildRunAction$1.run(BuildRunAction.java:91)
at java.lang.Thread.run(Thread.java:595)
[17:56:49]: [Updating sources] Patch was finished unexpectedly, can be found in file: C:\ba\temp\cache\temp35153patch70787
[17:56:49]: [Updating sources] Error while applying patch: null
[17:56:49]: Build finished
Case 2:
[14:44:00]: Changes are already collected
[14:44:01]: Clearing temporary directory: C:\ba\tp\buildTmp
[14:44:01]: Checkout directory: C:\ba\wk\7c3ade4f757b40b3
[14:44:01]: Updating sources (27s)
[14:44:00]: [Updating sources] Will perform clean checkout
[14:44:00]: [Updating sources] Clean checkout reasons
[14:44:00]: [Clean checkout reasons] The project sources on the agent are newer than requested
[14:44:00]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[14:44:00]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[14:44:24]: [Updating sources] Building incremental patch over the cached patch
[14:44:27]: [Updating sources] Building clean patch for VCS root: https://svn/Tools/tags/2009-Q3_1_RC1
[14:44:27]: [Updating sources] Failed to build patch for build #52, build id: 69239, VCS root: https://svn/Tools/tags/2009-Q3_1_RC1, due to error: org.tmatesoft.svn.core.SVNException: svn: Target path does not exist svn: REPORT of '/Tools/!svn/vcc/default': 500 Internal Server Error (https://svn)
[14:44:29]: [Updating sources] Repository sources transferred: 126.55Mb total
[14:44:29]: [Updating sources] Average transfer speed: 4.56Mb per second
[14:44:29]: Will repeat attempt when server will be available, number of attempts left: 2
[14:44:39]: Updating sources (33s)
[14:44:38]: [Updating sources] Will perform clean checkout
[14:44:38]: [Updating sources] Clean checkout reasons
[14:44:38]: [Clean checkout reasons] The project sources on the agent are newer than requested
[14:44:38]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[14:44:38]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[14:45:07]: [Updating sources] Building incremental patch over the cached patch
[14:45:11]: [Updating sources] Building clean patch for VCS root: https://svn/Tools/tags/2009-Q3_1_RC1
[14:45:11]: [Updating sources] Failed to build patch for build #52, build id: 69239, VCS root: https://svn/Tools/tags/2009-Q3_1_RC1, due to error: org.tmatesoft.svn.core.SVNException: svn: Target path does not exist svn: REPORT of '/Tools/!svn/vcc/default': 500 Internal Server Error (https://svn)
[14:45:12]: [Updating sources] Repository sources transferred: 126.55Mb total
[14:45:12]: [Updating sources] Average transfer speed: 3.77Mb per second
[14:45:13]: Will repeat attempt when server will be available, number of attempts left: 1
[14:45:23]: Updating sources (32s)
[14:45:22]: [Updating sources] Will perform clean checkout
[14:45:22]: [Updating sources] Clean checkout reasons
[14:45:22]: [Clean checkout reasons] The project sources on the agent are newer than requested
[14:45:22]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[14:45:22]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[14:45:50]: [Updating sources] Building incremental patch over the cached patch
[14:45:54]: [Updating sources] Building clean patch for VCS root: https://svn/Tools/tags/2009-Q3_1_RC1
[14:45:54]: [Updating sources] Failed to build patch for build #52, build id: 69239, VCS root: https://svn/Tools/tags/2009-Q3_1_RC1, due to error: org.tmatesoft.svn.core.SVNException: svn: Target path does not exist svn: REPORT of '/Tools/!svn/vcc/default': 500 Internal Server Error (https://svn)
[14:45:55]: [Updating sources] Repository sources transferred: 126.55Mb total
[14:45:55]: [Updating sources] Average transfer speed: 3.92Mb per second
[14:45:55]: [Updating sources] Patch is broken, can be found in file: C:\ba\tp\cache\temp13302patch69239
[14:45:55]: [Updating sources] Error while applying patch: Failed to build patch for build #52, build id: 69239, VCS root: https://svn/...
[14:45:54]: Build finished
Case 3:
[13:16:12]: Checking for changes
[13:16:14]: Clearing temporary directory: C:\ba\tp\buildTmp
[13:16:14]: Checkout directory: C:\ba\wk\411eec35e9d0cc98
[13:16:14]: Updating sources (2m:51s)
[13:16:12]: [Updating sources] Will perform clean checkout
[13:16:12]: [Updating sources] Clean checkout reasons
[13:16:12]: [Clean checkout reasons] Checkout directory is empty or doesn't exist
[13:16:12]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[13:16:12]: [Updating sources] Building clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[13:16:36]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[13:17:06]: [Updating sources] Building clean patch for VCS root: https://svn/Tools/branches/2009-Q3_X
[13:19:05]: [Updating sources] Problem while loading patch data stream: chunked stream ended unexpectedly
[13:19:05]: Will repeat attempt when server will be available, number of attempts left: 2
[13:19:15]: Updating sources (24m:39s)
[13:19:13]: [Updating sources] Will perform clean checkout
[13:19:13]: [Updating sources] Clean checkout reasons
[13:19:13]: [Clean checkout reasons] Checkout directory is empty or doesn't exist
[13:19:13]: [Clean checkout reasons] Agent doesn't have any version of the project sources
[13:19:13]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Source/Applications/branches/work_security
[13:21:03]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Tools/branches/2009-Q3_X
[13:21:03]: [Updating sources] Transferring cached clean patch for VCS root: https://svn/Tools/branches/2009-Q3_X
[13:21:04]: [Updating sources] Failed to build patch for build #10, build id: 68668, VCS root: https://svn/Tools/branches/2009-Q3_X, due to error: ClientAbortException: java.net.SocketException: Broken pipe
[13:21:15]: [Updating sources] Transferring repository sources: 178.59Mb so far...
[13:23:15]: [Updating sources] Transferring repository sources: 617.16Mb so far...
[13:25:15]: [Updating sources] Transferring repository sources: 934.31Mb so far...
[13:27:15]: [Updating sources] Transferring repository sources: 1279.66Mb so far...
[13:29:05]: [Updating sources] Repository sources transferred: 1605.37Mb total
[13:29:06]: [Updating sources] Average transfer speed: 2.71Mb per second
[13:33:20]: [Updating sources] Removing C:\ba\wk\411eec35e9d0cc98
[13:33:20]: [Updating sources] Updating C:\ba\wk\411eec35e9d0cc98
[13:34:17]: [Updating sources] Updating C:\ba\wk\411eec35e9d0cc98\Tools
[13:43:55]: Cannot find build file by path specified in build configuration settings: 'C:\ba\wk\411eec35e9d0cc98\Applications\Internal\Reference\ant_build.xml' (absolute path on agent). Please check that specified path is correct.
[13:43:55]: jetbrains.buildServer.RunBuildException: Cannot find build file by path specified in build configuration settings: 'C:\ba\wk\411eec35e9d0cc98\Applications\Internal\Reference\ant_build.xml' (absolute path on agent). Please check that specified path is correct.
jetbrains.buildServer.RunBuildException: Cannot find build file by path specified in build configuration settings: 'C:\ba\wk\411eec35e9d0cc98\Applications\Internal\Reference\ant_build.xml' (absolute path on agent). Please check that specified path is correct.
at jetbrains.buildServer.runner.BuildFileRunnerUtil.getBuildFile(BuildFileRunnerUtil.java:61)
at jetbrains.buildServer.agent.ant.AntBuildRunner.getBuildFile(AntBuildRunner.java:143)
at jetbrains.buildServer.agent.ant.AntBuildRunner.getProgramParameters(AntBuildRunner.java:81)
at jetbrains.buildServer.agent.runner.JavaProgramRunner.buildCommandLine(JavaProgramRunner.java:81)
at jetbrains.buildServer.agent.runner.GenericProgramRunner.run(GenericProgramRunner.java:103)
at jetbrains.buildServer.agent.impl.runner.adapt.BuildProcessImpl$2.run(BuildProcessImpl.java:55)
at java.lang.Thread.run(Thread.java:595)
[13:43:54]: Build finished
/J
Please sign in to leave a comment.
Hi,
I'm sorry for delay with the answer. It looks like these problems are caused by communication problems between SVN server and TeamCity server. Most likely, SVN server is overloaded and drops connections at some times. The overload can be caused by TeamCity itself, if it has many VCS roots and poll interval for SVN server is default one.
I'd reccomend to set Default VCS changes check interval in TeamCity to 5 minutes and to see if this helps (this can be done on the Server configuration page in Administration area).
Kind regards,
KIR
Hi Pavel,
I do encounter some svn: "chunked stream ended unexpectedly" error messages also (case 1).
For my team, 5 minutes for polling can't be a suitable solution.
Is there a corrective way?
Olivier.
Here are some more info about that problem.
It seems that set polling period to 5 minutes doesn't solve the problem.
For external parameter, in VCS root config, if set to "checkout but ignore change" instead of full support, there is no more connection problem with "chunked stream..." but all files are not loaded in agent working space... So maybe the problem is "SVN externals"-relative.
Olivier.
Hello Oliver,
Could you please take a look at the logs of your SVN server? May be there are some error messages regarding connectivity problems with TeamCity?
Regards,
KIR
Hi Pavel,
I'm investigating for one day on this problem...
I think that the problem comes from the SVN external feature support as it seems to fail when handling "some" links.
The problem happens almost all the time, but it could work one time, when a configuration change (like the subversion user, or the subversion URL in the TeamCity configuration), and then always fail again at next launch!
I've made my configuration work replacing some external links with file elements. Poor solution!
Difficult to say if the cause is from Subversion server (from Bitnami Redmine package) or from TeamCity plug-in.
Note that with TortoiseSVN client it works perfectly.
The message on server (Apache error.log) is :
Provider encountered an error while streaming a REPORT response. [500, #0]
A failure occurred while driving the update report editor [500, #160005]
Cannot replace a directory from within [500, #160005]
So is it that TeamCity asked for something wrong (like for ClearCase symlink support)?
If some one has ideas about that ...
Regards,
Olivier.
Hello Olivier,
Did you consider using "Checkout on Agent" option? This should reduce the load of the server during checkout.
The support of externals in TeamCity is somewhat heavy, and your problem can definetely be caused by externals support. But still, as I understand, this is subversion server which drops the connection in your case (messages like 'Broken pipe', 'chunked stream ended unexpectedly' witness to this).
Also, I noticed you're using TeamCity 4.5. I strongly suggest to try latest version of TeamCity, 4.5.5, because there were multiple enhancements in Svn support, and base Subversion support library was upgraded.
Regards,
KIR
Hi Pavel!
Thanks for your response.
Yes I do already use the "Checkout on Agent" option, and for performance reasons (as we already discussed previously on this forum).
The problem seems to focus on SVN external links support on file (not on folder). This should be supported since SVN 1.6.0.
My team uses AnkhSVN and TortoiseSVN in 1.6.6 version.
TeamCity 4.5.4 seems to use SVNKit 1.3.0, it's the same for TeamCity 5.0 EAP.
Do you really think that TeamCity 4.5.5 will perform best with externals?
Regards,
Olivier.
I've reproduced it.
I can confirm that adding a new link on a file with TortoiseSVN 1.6.6 makes problem in TeamCity 4.5.4 with VCS connection: "chunked stream ended unexpectedly".
Hello,
TeamCity 4.5.x doesn't support externals for files. This feature will be available in TC 5.0 only (it is already implemented).
Sorry
Regards,
KIR
Hi Pavel,
I've tried to upgrade TeamCity 4.5.4 with SVNKit 3.1.0 and with TeamCity 4.5.5 or 5.0 EAP for its SVN part.
No problem with SVNKit 3.1.0. It works the same, so with the problem with externals on files.
With SVNKit 3.1.0 and files from TeamCity 4.5.5, same behaviour.
With SVN files from TeamCity 5.0 EAP, TeamCity 4.5.4 can't work as problems:
- on agent part: "java.lang.NoSuchMethodError: jetbrains.buildServer.util.StringUtil.removeTailingSlash(Ljava/lang/String;)Ljava/lang/String;"
already known before on this forum, so build failure.
- on server part: "gins.spring.SpringPluginLoader - Failed to initialize spring context for plugin svn
org.springframework.beans.factory.CannotLoadBeanClassException: Error loading class [jetbrains.buildServer.buildTriggers.vcs.svn.SvnPersonalSupport] for bean with name 'svnPersonalSupport' defined in URL [file:/E:/Data/TeamCity/BuildServer/temp/spring-svn40279-spring.config.tmp.xml]: problem with class file or dependent class; nested exception is java.lang.NoClassDefFoundError: jetbrains/buildServer/vcs/IncludeRuleBasedMappingProvider"
so the plug-in isn't known any more
Any ideas?
Olivier.
Hi Olivier,
You cannot put SVN support from 5.0 to TeamCity 4.5.x. This won't work. You have either upgrade to 5.0 version entirely, or use 4.5.5 version as is.
Regards,
KIR
Hello,
Just to confirm - 4.5.5 doesn't support file externals. Latest 5.0 EAP does support.
And you cannot extract SVN support from 5.0 and put it to 4.5.5 to get file externals support in 4.5.5.
Regards,
KIR
Thanks for your help Kirill!
For now we had supressed file externals.
Maybe we will swich to TC 5.0 in the future.
Best regards,
Olivier.
very nice, saved us a day!