Getting 'TfsNativeExeRunner' exceptions randomly on proven builds

Just about every day I'm getting exceptions, log shown below, on various builds.  I'm using TFS as our source control.  If I retrigger the build it will work fine.

Please provide some assistance. (I've '@'d out server names and users)

Thanks

Dave

jetbrains.buildServer.vcs.VcsException: TFS execution error:
Stdout:
TFS Native Verifier v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Use Tfs from JetBrains.TeamCity.Tfs.Tfs9Accessor
TFS Native Accessor v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Connecting to server http://@@@@@@@:8080/
ERROR - TF14061: The workspace TeamCity-2834e2f0bf4440b1b002a71eef1423fe;@@\@@@@@@ does not exist.

Stderr:

Exception:
null
Exit code: 1
jetbrains.buildServer.vcs.VcsException: TFS execution error:
Stdout:
TFS Native Verifier v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Use Tfs from JetBrains.TeamCity.Tfs.Tfs9Accessor
TFS Native Accessor v4.0 Copyright (C) 2006-2009 JetBrains s.r.o.
INFO - Connecting to server http://@@@@@@@:8080/
ERROR - TF14061: The workspace TeamCity-2834e2f0bf4440b1b002a71eef1423fe;@@\@@@@@@@@ does not exist.

Stderr:

Exception:
null
Exit code: 1
at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsNativeExeRunner.start(TfsNativeExeRunner.java:43)
at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsServerNativeExeRunner.start(TfsServerNativeExeRunner.java:90)
at jetbrains.buildServer.buildTriggers.vcs.tfs.TfsUpdateByCheckoutRules.updateSources(TfsUpdateByCheckoutRules.java:31)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent.checkoutSources(ProjectSourcesOnAgent.java:61)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.checkoutSources(ProjectSourcesGetter.java:269)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesGetter.execute(ProjectSourcesGetter.java:123)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doGetSources(CheckoutSourcesStage.java:45)
at jetbrains.buildServer.agent.impl.runStages.CheckoutSourcesStage.doRecoverableStage(CheckoutSourcesStage.java:31)
at jetbrains.buildServer.agent.impl.runStages.RecoverableBuildStage.doBuildStage(RecoverableBuildStage.java:61)
at jetbrains.buildServer.agent.impl.BuildRunAction.doStages(BuildRunAction.java:135)
at jetbrains.buildServer.agent.impl.BuildRunAction.access$000(BuildRunAction.java:21)
at jetbrains.buildServer.agent.impl.BuildRunAction$1.run(BuildRunAction.java:91)
at java.lang.Thread.run(Thread.java:595)

49 comments
Comment actions Permalink

Could you please check there is no build configurations with custom checkout directory.

0
Comment actions Permalink

There is only one out of the 48 configurations that have a custom directory.  And it does fail.

0
Comment actions Permalink

Setting custom checkout directory may cause several builds to be running in the same working directory if there are 2+ build agents running on the same machine. There will be 2+ concurrently running TFS checkouts targeted to the same directory and this will cause a error in TeamCity TFS implementation.

Is that possible to use default checkout directory for that build configuration? If not, please make it be compatible only with one build agent per machine.

0
Comment actions Permalink

I don't believe that the build with the custom directory would be a problem.  It has only failed twice and these builds now fail when they are the only build running on any build box.  I can't use the default directory for this build definition, the paths become too long.  In fact each this custom directory uses an agent variable 'AgentID' that I put on each agent install to distinguish each agent from all the ohers in a case just like this.  So, even the custom directory is unique for each agent.

Putting a single build agent per machine is very counter productive.  We've spent money on 17 agents (plus the 3 already) for 5 build boxes and now you're saying I have can't put multiple agents on the same box?  That is not a possible.  I would have to increase my machine count to get usage out of all the agents.  I'd be better off going back to TFS' build system that can handle multiple agents per machine.

Are you saying this can't be fixed to work like any other SCM TeamCity supports?

0
Comment actions Permalink

Dave, It's enough to use AgentID in custom checkout directories for build configurations.
There is no need to deny running several build agents on one PC.
I'll re-check TeamCity TFS utility and try to reproduce your case again.

Thanks!

0
Comment actions Permalink

How much VCS roots do you use in build configurations?

0
Comment actions Permalink

I'm not sure I understand the question.

But each build has at least one VCS root and usually 2.  If that answers the question.

0
Comment actions Permalink

Yes, right. Thank you.

0
Comment actions Permalink

I reproduced one more bug. Could you please
apply new patch from this post.
To apply it, please do the following:
- stop TeamCity server
- open <server>/webapps/ROOT/WEB-INF/plugins/
- backup tfs folder
- replace it's contents with contents of attached tfs-9046.zip
- start server.

Build agents should autoupgrade in a short while.

Please let me know if the patch helped.
Thanks!



Attachment(s):
tfs-9046.zip
0
Comment actions Permalink

Thanks!  I'll do ASAP!

I'll let you know the results.

0
Comment actions Permalink

I have one build that failed.  Looks similar to the others.  When I went to look up the workspace it failed to create 'cause the VCS already exists in another workspace, it didn't exist.  Log attached.



Attachment(s):
TeamCityFailure.txt
0
Comment actions Permalink

Thank you. I'll analize your log in more detail. Could you please remove all TeamCity-A8* workspaces from that machine.
Do you connect all build agents from the machine to the same TeamCity server?

Does the worksaround help?

Thanks!

0
Comment actions Permalink

I have removed all TeamCity workspaces.  There is only one TeamCity server controlling all the agents.

I will be testing this shortly.

0
Comment actions Permalink

Failed.  I attached the log file.

It fails when it is the only build running.  I had to change the agent for it to build correctly.

I had far more failures.  Any build that tried to build on the agent it previously built on failed.



Attachment(s):
TeamCityFailure.txt
0
Comment actions Permalink

I'm also getting increased failures when labeling.  In this case the label is: SUMCollectorWebConsole-%system.build.number%

Attached is the exception thrown from within the email sent.

Thanks.



Attachment(s):
FailingToLabel.txt
0
Comment actions Permalink

Hello,
I've created an issue for the latest request at http://www.jetbrains.net/tracker/issue2/TW-8669
Please watch/vote for this.

I've added TFS checkout error at http://www.jetbrains.net/tracker/issue2/TW-8670
Please watch/vote for this too.

0
Comment actions Permalink

So, now I have to wait for the next release or so for this to be corrected?

0
Comment actions Permalink

I'll check TFS for that fantom workspaces that was not found in the request to TFS. Could you please check that this agent uses same tfs-native.exe as was attached to my recent post. I've also tried to reproduce your case with 4 running build agent on one machine and got no errors. But before the try I had to remove all TeamCity-* workspaces before the test. Propably this will solve you issue.

The latest SQL related error is assigned to another developer. Please watch this request for updates and/or questions. Please attach zipped logs folder from your TeamCity server.

Dave, let's continue our discussion in the tracker.

0
Comment actions Permalink

I'll check to make sure all the agents are using the correct TFS-native.exe.  Before I used your latest update I removed all the teamcity-* workspace from our TFS server.  That's when the errors became worse then ever during this trial.  I also waited for all the agents to update themselves.

After this post, I'll keep it in the tracker and let you know if I do find any wrong versioned TFS-native.exe on any agent.

Dave

0

Please sign in to leave a comment.