Getting 'TfsNativeExeRunner' exceptions randomly on proven builds

What version of the server is it?

Does anybody cleaning workspaces in TFS for that user?
Do you run builds simultaniously on several agents?

Could you please check if the error is caused by 2+ builds that updates sources

Thanks!

#1) I'm using 4.5.0 Build 8944 of TeamCity

#2) I'm not sure what you mean...

#3) Yes, these build are distributed across 20 possible agents and I haven't found an agent relation ship.

#4) If i understand, I check a couple and the log says:[06:54:16]: [Updating sources] Updating project sources (6s)

Thanks for your time.

Dave

Do you have several build agents running on the same machine?

Yes.  They all run in separate subdirectories, ports etc.  For what it's worth, I didn't experience this until after the 4.5.0 upgrade.

Sorry, I'm not sure I got the last sentence in the right way. Do you still have the issue reproducing?

TeamCity build agent TFS checkout may fail if running the several build agents on the same machine. This is because
TFS workspace will be reused between those agents and it may cause errors when two build agents are running
TFS checkout at the same time.

I'll add a workaround for the issue.

Yes.

But there is never the same build running at the same time, if I understand your response.  Wouldn't each build have a separate workspace for each box?  In our setup, you'd never have build 'ABC' running on two separate agents on the same box.

The issue could happen even if there are differend build configurations started with TFS checkout on the agent.

TeamCity TFS checkout tool uses guid in workspace name.
But, it tries to reuse allready created workspaces too.

I think I can add build agent name to the workspace name in order to reduce collisions.

Do you need a patch for this?

I would greatly appreciate it.  It would take away a lot of the late night build worries.

Thanks.

Could you please try TeamCity 4.5.2

Thank you very much.  I'll install it and let you know.

Dave

I did the 'Build Action' to clean all the agents.  I'm still getting the same problem.  The log is much longer showing warnings in addition to the original exception. Typical log file attached.

Something that is different this time...  Before, if I triggered the build by itself, it would work fine.  Now if I trigger another build and it uses the same agent it fails.  If, after a successful build on a different agent, it will some times succeed on the previous agent that failed.  Those that don't succeed, I don't know how to get it to successfully build on that agent. These agents in this trial are on the same build box.

Dave

I've changed the naming for workspaces. So each build agent will create it's own TFS workspace. In logs you see warnings from the tool when it tries to remove old TeamCity workspaces, that was named like: TeamCity-[0-9a-f]+. All new workspaces are named like: TeamCity-[a-z0-9]+-[a-z0-9]+.

I believe running only one build at a time will allow TeamCity to remove older workspaces. On the other hand, you may try deleting all old workspaces explicitly in TeamExplorer. (You may simply delete all workspaces with name starting from TeamCity)

Thanks!

Thanks, I'll give that a try.

Dave

I cleaned out all the TeamCity* workspaces and I'm still getting exceptions.

Please advise.

Dave, do you use the same user account for TFS? Have you recently changed it?

Have you recently changed checkout rules? What are the current checkout rules, what are the TFS mapping for TeamCity-A8-2n3cgptjsgv22f50tfzc2xrv22

Yes the same account.

I have not changed any checkout rules.  The workspace TeamCity-A8-2n3cgptjsgv22f50tfzc2xrv22 doesn't exist.

The checkout rules for each VCS contain only one item.  For example if I have a TFS path of:

$/Project/main/SubProject/Something

The rule will usually be:

+:. => SubProject/Something

I reproduced the issue and created a patch for you.
Please do the following:
- stop server

To patch server TFS checkout:
- open <server>/webapps/ROOT/WEB-INF/plugins/tfs/server/tfs-server.jar
- backup this file
- replace /bin folder contents with a patch from this post

To patch agent TFS checkout
- open <server>/webapps/ROOT/WEB-INF/plugins/tfs/agent/tfs-agent.zip
- backup tfs-agent.zip
- replace tfs-agent/bin folder contents with a patch from this forum post

- start server

Thanks!

Where are the patch files?

Sorry, I've forgotten to attach the file to you.
Please let me know the result.

The zip file only has tfs-native.exe and tfs-native.pdb files.  No .jar file.

Dave, I posted the instruction how to apply that .exe file to you TeamCity installation.
I've just added the easier patch.

To apply the patch form that issue, please do the following:
- stop server
- open <server>/webapps/ROOT/WEB-INF/plugins
- backup tfs folder
- replace it with new folder from attachement
- start server
- wait until all build agents autoupgrade

I suppose this way would be easier.
Thanks.

Installed code as instructed to do.  After agents came back online, cleaned all agents.  Out of 20 builds I still got two builds with the attached error.

The two builds would not let me build on the same build on a different agent on the same build machine.  I had to go to a different build box.  Also double checked to make sure tfs-agent.exe was 9031 build, and they were.

Dave

Dave, could you please check work directories of the build agent does not point to the same working directory.
Please remove those workspaces as workaround.
Post the mentioned workspace TeamCity-A0-* mappings. Are there any other TeamCity-A0-* workspaces. Does they mapped to the same folder?

Thanks!

None of the agents have the same working directory, I checked their respective configurations/properties.  Attached are all the TeamCity-A0* workspaces. I'll delete all of them and try again.  When ever I look into the exception, i try to find the workspaces mentioned and I never find those workspaces.

Dave

Additional bit of information...

I appears the builds I've looked at this morning share a common VCS root.

Has this thread stopped?  I'm still working with hit and miss builds...

Dave

Dave, I was on vocation for those days. I'll reply you today. Sorry.

Not a problem.  I've been keeping on the failed builds.  I hope there's enough additional information.