Getting 'TfsNativeExeRunner' exceptions randomly on proven builds

Just about every day I'm getting exceptions, log shown below, on various builds.  I'm using TFS as our source control.  If I retrigger the build it will work fine.

Please provide some assistance. (I've '@'d out server names and users)

Thanks

Dave

jetbrains.buildServer.vcs.VcsException: TFS execution error:
Stdout:
TFS Native Verifier v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Use Tfs from JetBrains.TeamCity.Tfs.Tfs9Accessor
TFS Native Accessor v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Connecting to server http://@@@@@@@:8080/
ERROR - TF14061: The workspace TeamCity-2834e2f0bf4440b1b002a71eef1423fe;@@\@@@@@@ does not exist.

Stderr:

Exception:
null
Exit code: 1
jetbrains.buildServer.vcs.VcsException: TFS execution error:
Stdout:
TFS Native Verifier v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Use Tfs from JetBrains.TeamCity.Tfs.Tfs9Accessor
TFS Native Accessor v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Connecting to server http://@@@@@@@:8080/
ERROR - TF14061: The workspace TeamCity-2834e2f0bf4440b1b002a71eef1423fe;@@\@@@@@@@@ does not exist.

Stderr:

Exception:
null
Exit code: 1
at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsNativeExeRunner.start(TfsNativeExeRunner.java:43)
at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsServerNativeExeRunner.start(TfsServerNativeExeRunner.java:90)
at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsUpdateByCheckoutRules.updateSources(TfsUpdateByCheckoutRules.java:31)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent.checkoutSources(ProjectSourcesOnAgent.java:61)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.checkoutSources(ProjectSourcesGetter.java:269)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.execute(ProjectSourcesGetter.java:123)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doGetSources(CheckoutSourcesStage.java:45)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doRecoverableStage(CheckoutSourcesStage.java:31)
at jetbrains.buildServer.agent.impl.runStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:61)
at jetbrains.buildServer.agent.impl.BuildRunAction.doStages(BuildRunAction.java:135)
at jetbrains.buildServer.agent.impl.BuildRunAction.access$000(BuildRunAction.java:21)
at jetbrains.buildServer.agent.impl.BuildRunAction$1.run(BuildRunAction.java:91)
at java.lang.Thread.run(Thread.java:595)

49 comments
Comment actions Permalink

What version of the server is it?

Does anybody cleaning workspaces in TFS for that user?
Do you run builds simultaniously on several agents?

Could you please check if the error is caused by 2+ builds that updates sources

Thanks!

0
Comment actions Permalink

#1) I'm using 4.5.0 Build 8944 of TeamCity

#2) I'm not sure what you mean...

#3) Yes, these build are distributed across 20 possible agents and I haven't found an agent relation ship.

#4) If i understand, I check a couple and the log says:[06:54:16]: [Updating sources] Updating project sources (6s)

Thanks for your time.

Dave

0
Comment actions Permalink

Do you have several build agents running on the same machine?

0
Comment actions Permalink

Yes.  They all run in separate subdirectories, ports etc.  For what it's worth, I didn't experience this until after the 4.5.0 upgrade.

0
Comment actions Permalink

Sorry, I'm not sure I got the last sentence in the right way. Do you still have the issue reproducing?

TeamCity build agent TFS checkout may fail if running the several build agents on the same machine. This is because
TFS workspace will be reused between those agents and it may cause errors when two build agents are running
TFS checkout at the same time.

I'll add a workaround for the issue.

0
Comment actions Permalink

Yes.

But there is never the same build running at the same time, if I understand your response.  Wouldn't each build have a separate workspace for each box?  In our setup, you'd never have build 'ABC' running on two separate agents on the same box.

0
Comment actions Permalink

The issue could happen even if there are differend build configurations started with TFS checkout on the agent.

TeamCity TFS checkout tool uses guid in workspace name.
But, it tries to reuse allready created workspaces too.

I think I can add build agent name to the workspace name in order to reduce collisions.

Do you need a patch for this?

0
Comment actions Permalink

I would greatly appreciate it.  It would take away a lot of the late night build worries.

Thanks.

0
Comment actions Permalink

Could you please try TeamCity 4.5.2

0
Comment actions Permalink

Thank you very much.  I'll install it and let you know.

Dave

0
Comment actions Permalink

I did the 'Build Action' to clean all the agents.  I'm still getting the same problem.  The log is much longer showing warnings in addition to the original exception. Typical log file attached.

Something that is different this time...  Before, if I triggered the build by itself, it would work fine.  Now if I trigger another build and it uses the same agent it fails.  If, after a successful build on a different agent, it will some times succeed on the previous agent that failed.  Those that don't succeed, I don't know how to get it to successfully build on that agent. These agents in this trial are on the same build box.

Dave



Attachment(s):
TC4.5.2-Exception.txt
0
Comment actions Permalink

I've changed the naming for workspaces. So each build agent will create it's own TFS workspace. In logs you see warnings from the tool when it tries to remove old TeamCity workspaces, that was named like: TeamCity-[0-9a-f]+. All new workspaces are named like: TeamCity-[a-z0-9]+-[a-z0-9]+.

I believe running only one build at a time will allow TeamCity to remove older workspaces. On the other hand, you may try deleting all old workspaces explicitly in TeamExplorer. (You may simply delete all workspaces with name starting from TeamCity)

Thanks!

0
Comment actions Permalink

Thanks, I'll give that a try.

Dave

0
Comment actions Permalink

I cleaned out all the TeamCity* workspaces and I'm still getting exceptions.

Please advise.



Attachment(s):
TC4.5.2-Exceptions2.txt
0
Comment actions Permalink

Dave, do you use the same user account for TFS? Have you recently changed it?

0
Comment actions Permalink

Have you recently changed checkout rules? What are the current checkout rules, what are the TFS mapping for TeamCity-A8-2n3cgptjsgv22f50tfzc2xrv22

0
Comment actions Permalink

Yes the same account.

0
Comment actions Permalink

I have not changed any checkout rules.  The workspace TeamCity-A8-2n3cgptjsgv22f50tfzc2xrv22 doesn't exist.

The checkout rules for each VCS contain only one item.  For example if I have a TFS path of:

$/Project/main/SubProject/Something

The rule will usually be:

+:. => SubProject/Something

0
Comment actions Permalink

I reproduced the issue and created a patch for you.
Please do the following:
- stop server

To patch server TFS checkout:
- open <server>/webapps/ROOT/WEB-INF/plugins/tfs/server/tfs-server.jar
- backup this file
- replace /bin folder contents with a patch from this post

To patch agent TFS checkout
- open <server>/webapps/ROOT/WEB-INF/plugins/tfs/agent/tfs-agent.zip
- backup tfs-agent.zip
- replace tfs-agent/bin folder contents with a patch from this forum post

- start server

Thanks!

0
Comment actions Permalink

Where are the patch files?

0
Comment actions Permalink

Sorry, I've forgotten to attach the file to you.
Please let me know the result.



Attachment(s):
tfs-native-9031.zip
0
Comment actions Permalink

The zip file only has tfs-native.exe and tfs-native.pdb files.  No .jar file.

0
Comment actions Permalink

Dave, I posted the instruction how to apply that .exe file to you TeamCity installation.
I've just added the easier patch.

To apply the patch form that issue, please do the following:
- stop server
- open <server>/webapps/ROOT/WEB-INF/plugins
- backup tfs folder
- replace it with new folder from attachement
- start server
- wait until all build agents autoupgrade

I suppose this way would be easier.
Thanks.



Attachment(s):
tfs-9031.zip
0
Comment actions Permalink

Installed code as instructed to do.  After agents came back online, cleaned all agents.  Out of 20 builds I still got two builds with the attached error.

The two builds would not let me build on the same build on a different agent on the same build machine.  I had to go to a different build box.  Also double checked to make sure tfs-agent.exe was 9031 build, and they were.

Dave



Attachment(s):
TC4.5.2-Exception3.txt
0
Comment actions Permalink

Dave, could you please check work directories of the build agent does not point to the same working directory.
Please remove those workspaces as workaround.
Post the mentioned workspace TeamCity-A0-* mappings. Are there any other TeamCity-A0-* workspaces. Does they mapped to the same folder?

Thanks!

0
Comment actions Permalink

None of the agents have the same working directory, I checked their respective configurations/properties.  Attached are all the TeamCity-A0* workspaces. I'll delete all of them and try again.  When ever I look into the exception, i try to find the workspaces mentioned and I never find those workspaces.

Dave



Attachment(s):
TeamCity-A0.txt
0
Comment actions Permalink

Additional bit of information...

I appears the builds I've looked at this morning share a common VCS root.

0
Comment actions Permalink

Has this thread stopped?  I'm still working with hit and miss builds...

Dave

0
Comment actions Permalink

Dave, I was on vocation for those days. I'll reply you today. Sorry.

0
Comment actions Permalink

Not a problem.  I've been keeping on the failed builds.  I hope there's enough additional information.

0

Please sign in to leave a comment.