Problems getting project sources using SVN with externals

Hi,

Lately we have gotten some intermittently gotten this error message from out Team City server:

Could not get project sources: for build #679

This is our setup:

  • We are using Subversion as our VSC
  • We are running teamcity v. 4.0.2.
  • Checkout is on server using default directory
  • Buildagent is running on same machine as server


In our subversion repository we do have references to external subversion repositories (svn:external). These repos does not have the same revision counter, so forinstance the rev 10197 mentioned below in the error does not exist.
The svn;external is rev locked to rev 55 - so I do not understand why it is trying to checkout anything regarding rev 10197 from our 3rdParty repo


Any help on this is greatly appreciated - also note that this only happens sometimes, but when it happens it seems to happen continously until I restart the server.


Olav Kindt

------
Here is a complete build log for one of the times it has failed:
---
[10:34:31]: bt2
[10:34:31]: Checking for changes
[10:34:32]: Clean build enabled: removing old files from D:\TeamCity\BuildAgent1\work\1e4e48170e22602a
[10:34:32]: Clearing temporary directory: D:\TeamCity\BuildAgent1\temp\buildTmp
[10:34:32]: Checkout directory: D:\TeamCity\BuildAgent1\work\1e4e48170e22602a
[10:34:32]: Getting project sources
[10:34:36]: Could not get project sources: for build #679
[10:34:36]: jetbrains.buildServer.vcs.VcsException: Getting sources for checkout rule '.=>.' failed with erorr: org.tmatesoft.svn.core.SVNException: svn: No such revision 10197
svn: REPORT of '/svn/3rdParty/!svn/vcc/default': 500 Internal Server Error (http://svn:8080)
jetbrains.buildServer.vcs.VcsException: Getting sources for checkout rule '.=>.' failed with erorr: org.tmatesoft.svn.core.SVNException: svn: No such revision 10197
svn: REPORT of '/svn/3rdParty/!svn/vcc/default': 500 Internal Server Error (http://svn:8080)
    at jetbrains.buildServer.vcs.VcsSupportUtil.buildPatch(VcsSupportUtil.java:86)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.buildPatch(SvnSupport.java:271)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchCacheImpl.buildCleanPatch(PatchCacheImpl.java:31)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:16)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:127)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:83)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.buildPatch(BuildTypeImpl.java:333)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl$$FastClassByCGLIB$$a84db719.invoke(<generated>)
    at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
    at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
    at jetbrains.buildServer.serverSide.impl.auth.TeamCityMethodSecurityInterceptor.invoke(TeamCityMethodSecurityInterceptor.java:10)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:582)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl$$EnhancerByCGLIB$$d68247f0.buildPatch(<generated>)
    at jetbrains.buildServer.serverSide.impl.BuildStarter$2.call(BuildStarter.java:13)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
    at java.util.concurrent.FutureTask.run(FutureTask.java:123)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)
Caused by: jetbrains.buildServer.vcs.VcsException: org.tmatesoft.svn.core.SVNException: svn: No such revision 10197
svn: REPORT of '/svn/3rdParty/!svn/vcc/default': 500 Internal Server Error (http://svn:8080)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.buildPatch(SvnSupport.java:217)
    at jetbrains.buildServer.vcs.VcsSupportUtil.buildPatch(VcsSupportUtil.java:84)
    ... 20 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: No such revision 10197
svn: REPORT of '/svn/3rdParty/!svn/vcc/default': 500 Internal Server Error (http://svn:8080)
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1212)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:1076)
    at org.tmatesoft.svn.core.io.SVNRepository.update(SVNRepository.java:1932)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.exportFiles(SvnConnection.java:994)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.access$800(SvnConnection.java:37)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection$10.runAction(SvnConnection.java:1027)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.processAllExternals(SvnConnection.java:368)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.processExternalsExport(SvnConnection.java:1012)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.exportFiles(SvnConnection.java:999)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.buildPatch(SvnConnection.java:928)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection$6.runAction(SvnConnection.java:435)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.processAllExternals(SvnConnection.java:368)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.processExternalsPatch(SvnConnection.java:401)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.buildPatch(SvnConnection.java:956)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.buildPatch(SvnSupport.java:211)
    ... 21 more

[10:34:36]: Could not get project sources
[10:34:37]: Loading changed files from server...
[10:34:39]: Patch is broken, can be found in file D:\TeamCity\BuildAgent1\temp\cache\temp46972patch679
[10:34:39]: Error while applying patch: Getting sources for checkout rule '.=>.' failed with erorr: org.tmatesoft.svn.core.SVNException: ...
[10:34:39]: Build finished

18 comments
Comment actions Permalink

Hello Olav,

   Could you please verify that SVN repository UUIDs are different for the main repository and for the repository attached via svn:externals.
   The UUID could be obtained via command svn info .

   Please also provide the example of your svn:externals configuration.

   Please also provide details of your SVN settings, and collect debug logs as described at http://url.ie/12kd .

   Kind regards,
   KIR

0
Comment actions Permalink

Hi Kirill,

The SVN UUID are indeed different

Here is the relevant info :

Repo 1 - our main repo

C:\prg\MyApp\trunk $ svn info .
Path: .
URL: http://svn:8080/svn/Main/MyApp/trunk
Repository Root: http://svn:8080/svn/Main
Repository UUID: 451bf9ac-8036-db11-a3f3-000cf19ac343
Revision: 10199
Node Kind: directory
Schedule: normal
Last Changed Author: XXXXX\xx
Last Changed Rev: 9875
Last Changed Date: 2009-01-11 21:27:13 +0100 (so, 11 jan 2009)


Repo 2 - our 3rdParty component repo (doesn't change that often)

C:\prg\MyApp\trunk\Externals\External1 $ svn info .
Path: .
URL: http://svn:8080/svn/3rdParty/ExternalDll_1
Repository Root: http://svn:8080/svn/3rdParty
Repository UUID: f590ae36-c031-ec4a-bd2c-8280c637b480
Revision: 55
Node Kind: directory
Schedule: normal
Last Changed Author: XXXXX\xx
Last Changed Rev: 55
Last Changed Date: 2008-11-05 10:34:55 +0100 (on, 05 nov 2008)


Here is my svn:external (for MyApp/trunk)

buildtools http://svn:8080/svn/Main/shared/buildtools/trunk
SomeLib http://svn:8080/svn/Main/SomeLib/trunk/SomeLib
CoreLib http://svn:8080/svn/Main/shared/.Net/CoreLib/trunk/Source
CoreLibTests http://svn:8080/svn/Main/shared/.Net/CoreLib/trunk/Tests
LicensingLib http://svn:8080/svn/Main/shared/.Net/LicensingLib/trunk/Source
Externals/CommandLine http://svn:8080/svn/Main/shared/.Net/CommandLine/Source
Externals/External1 -r 55 http://svn:8080/svn/3rdParty/ExternalDll_1


I will have to get back to you on the last item (logs) since my "problem" now is that it currently works. I'll update this post when it starts failing again :-)


Olav

0
Comment actions Permalink

Olav, thanks a lot for the info, I'll try to test you configuration locally.

Kind regards,
KIR

Update: couldn't reproduce

0
Comment actions Permalink

Just to make matters worse, it now happens again, but with a different exception message : this time it fails with:

[12:21:41]: bt2
[12:21:41]: Checking for changes
[12:21:41]: Clean build enabled: removing old files from D:\temp
[12:21:41]: Clearing temporary directory: D:\TeamCity\BuildAgent1\temp\buildTmp
[12:21:41]: Checkout directory: D:\temp
[12:21:41]: Getting project sources
[12:21:48]: Could not get project sources: for build #783
[12:21:48]: jetbrains.buildServer.vcs.VcsException: Getting sources for checkout rule '.=>.' failed with erorr: org.tmatesoft.svn.core.SVNAuthenticationException: svn: Authentication required for '<http://svn:8080>'
jetbrains.buildServer.vcs.VcsException: Getting sources for checkout rule '.=>.' failed with erorr: org.tmatesoft.svn.core.SVNAuthenticationException: svn: Authentication required for '<http://svn:8080>'
    at jetbrains.buildServer.vcs.VcsSupportUtil.buildPatch(VcsSupportUtil.java:86)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.buildPatch(SvnSupport.java:271)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchCacheImpl.buildCleanPatch(PatchCacheImpl.java:31)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:16)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:127)
    at jetbrains.buildServer.serverSide.impl.projectSources.PatchComposer.buildPatch(PatchComposer.java:83)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.buildPatch(BuildTypeImpl.java:333)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl$$FastClassByCGLIB$$a84db719.invoke(<generated>)
    at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
    at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
    at jetbrains.buildServer.serverSide.impl.auth.TeamCityMethodSecurityInterceptor.invoke(TeamCityMethodSecurityInterceptor.java:10)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
    at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:582)
    at jetbrains.buildServer.serverSide.impl.BuildTypeImpl$$EnhancerByCGLIB$$7695eae9.buildPatch(<generated>)
    at jetbrains.buildServer.serverSide.impl.BuildStarter$2.call(BuildStarter.java:13)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
    at java.util.concurrent.FutureTask.run(FutureTask.java:123)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)
Caused by: jetbrains.buildServer.vcs.VcsException: org.tmatesoft.svn.core.SVNAuthenticationException: svn: Authentication required for '<http://svn:8080>'
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.buildPatch(SvnSupport.java:217)
    at jetbrains.buildServer.vcs.VcsSupportUtil.buildPatch(VcsSupportUtil.java:84)
    ... 20 more
Caused by: org.tmatesoft.svn.core.SVNAuthenticationException: svn: Authentication required for '<http://svn:8080>'
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:62)
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:597)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:257)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:245)
    at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:258)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1210)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.diff(DAVRepository.java:1056)
    at org.tmatesoft.svn.core.wc.SVNDiffClient.doDiffURLURL(SVNDiffClient.java:2882)
    at org.tmatesoft.svn.core.wc.SVNDiffClient.doDiff(SVNDiffClient.java:548)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.diffFiles(SvnConnection.java:969)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.buildPatch(SvnConnection.java:939)
    at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.buildPatch(SvnSupport.java:211)
    ... 21 more


[12:21:48]: Could not get project sources
[12:21:48]: Loading changed files from server...
[12:21:49]: Patch is broken, can be found in file D:\TeamCity\BuildAgent1\temp\cache\temp22538patch783
[12:21:49]: Error while applying patch: Getting sources for checkout rule '.=>.' failed with erorr: org.tmatesoft.svn.core.SVNAuthenticat...
[12:21:49]: Build finished




I have attached the logfiles:

   teamcity-svn.log
   teamcity-vcs.log
   teamcity-server.log


VCS Checkout mode is "Automatically on server"



Attachment(s):
TeamCity-logs.zip
0
Comment actions Permalink

Hello Olav,

  Do you have the same auth settings for main repository and for externals?

  Regards,
  KIR

0
Comment actions Permalink

Yes - the auth settings should be the same.

I have no problem checking out the source code using the same user with TortoiseSVN

Olav

0
Comment actions Permalink

Olav, did you do anything what may trigger the problem? Any change in SVN externals, in VCS Root settings, SVN server settings?

Regards,
KIR

0
Comment actions Permalink

No - none that I am aware of.

As you can see from this screenshot, it worked fine until build 296  which is the first point where it failes with the authentication error. Then it worked fine for 2 builds, and since then it has failed.
No changes has been done to any externals in this period, just "regular" code updates


TeamCity-Screenshot.png

0
Comment actions Permalink

Not sure whether this will help, but please try to restart your TeamCity server.

Regards,
KIR

0
Comment actions Permalink

I have already tried that - I shut down the Web Server, and the build agent, and brought them back - but no change.

I even tried clearing out all the code on the buildagent/work directory to force a clean checkout - but I'm not sure that is the "proper" method.

Is there a way for me to clear all cached work - and start up again (to force a full - clean project checkout?)

0
Comment actions Permalink

As I see from the logs, you're using NTLM authentication on the SVN server, correct?
If so, could you please try setting svnkit authentication property as described at http://url.ie/16qw

Regards,
KIR

0
Comment actions Permalink

Hi again Kirill,

Thanks for all your help so far.

I tried what you suggested, and there is no change. I did however try to connect with a different user account - and that seems to work for now. BUT, the user I previously were using did have access before (using TeamCity), and has OK access when checking out code manually (using TortoiseSVN) - so something is still a little fishy :-(


I will let it run for a few days using the second login, and see how that goes.

Olav

0
Comment actions Permalink

Hi Olav,

  I've found another evendence for the problem: http://jetbrains.net/tracker/issue/TW-6629
  Please watch/vote/comment for it.

  I will attach your logs to the issue, if you don't mind.

  Regards,
  KIR

0
Comment actions Permalink

Just some additional info.

I got it to work for a while by changing the user name of the SVN user (even though the old one had worked before), but now, after about 5 days, even the second user name has stopped working :-(

0
Comment actions Permalink

Hi Olav,

  Are you sure you've correctly passed -Dsvnkit.http.methods=Basic,NTLM option to the TeamCity server?
  If you run TeamCity as a windows service, this should be done in the service configuration, please see
  http://www.jetbrains.net/confluence/display/TCD4/Server+Startup+Properties

  Kind regards,
  KIR

0
Comment actions Permalink

Hi Kirill,

The builds are currently running ok (started running again after a few failed ones), and have been stable for a while.

Here is what I have in the Java Options

-Dsvnkit.http.methods=NTLM,Negotiate,Digest,Basic

Do you think I should remove the Negotiate and Digest settings?

0
Comment actions Permalink

Hi Olav,

  The point is to put NTLM at the end of the list, i.e. -Dsvnkit.http.methods=Negotiate,Digest,Basic,NTLM

  Looks like NTLM auth is pretty unstable in SVNKit.

   Kind regards,
   KIR

0
Comment actions Permalink

Hi again Kirill

I did what you said, and now I have gotten it to work for the user that previously didn't work.

Looking back at the logs, it seems that every 10th build failed (using the "good" user)

I will let it run for a couple of days now, and see if the "bad" user works consistently.

Thanks for your help

Olav

0

Please sign in to leave a comment.