git (jetbrains) - checkout failure (The remote end hung up unexpectedly)

We're using agent checkout for our compile and unit test builds, and see this error intermittently for some projects, and 100% of the time for other projects:

[17:41:32]: Checking for changes
[17:41:33]: Clearing temporary directory: C:\BuildAgent\temp\buildTmp
[17:41:33]: Checkout directory: C:\BuildAgent\work\ab090aa796030217
[17:41:33]: Updating sources: Agent side checkout... (41s)
[17:41:33]: [Updating sources: Agent side  checkout...] Will perform clean checkout. Reason: Agent doesn't have any  version of the project sources
[17:41:33]: [Updating sources: Agent side checkout...] Cleaning C:\BuildAgent\work\ab090aa796030217
[17:41:33]: [Updating sources: Agent side checkout...] VCS Root: ***NameHere*** Master (41s)
[17:41:33]: [VCS Root: ***NameHere*** Master] revision: 77fb84190d4251fcedeb933156c927213b4e5b14@12ec8e9e498
[17:41:33]: [VCS Root: ***NameHere*** Master] Cleaning C:\BuildAgent\work\ab090aa796030217
[17:41:33]: [VCS Root: ***NameHere*** Master] The .git directory is missing in  'C:\BuildAgent\work\ab090aa796030217'. Running 'git init'...
[17:41:34]: [VCS Root: ***NameHere*** Master] Fetching data for '***NameHere*** Master'...
[17:42:14]: [Updating sources: Agent side checkout...] Failed  to perform checkout on agent: 'c:\TeamCityBuildTools\git\bin\git.exe  fetch --no-tags -q origin +refs/heads/master:refs/remotes/origin/master'  command failed. stderr: fatal: The remote end hung up unexpectedly
[17:42:14]: Build finished


Bear in mind that this shouldn't be a permissions or configuration problem, as the clones succeed some of the time, which leads me to think its some kind of timeout or similar error. Amazingly, logging onto the buildagents as the same user and executing the identical commands (init, clone, fetch, etc) will succeed without fail.



Our software versions:
TeamCity Enterprise Version 5.1.3 (build 13506)
git 1.6.0.4
gitorious commit  4f084ffc34241564642781294842ad4228af017f (Fri Mar 18 15:12:26 2011)

sshd OpenSSH_5.1p1 Debian-5ubuntu1, OpenSSL 0.9.8g 19 Oct 2007


Any ideas on how we can go about resolving this?

10 comments
Comment actions Permalink

Hi Matt, what kind of authentication do you use?

0
Comment actions Permalink

Private Key

Most annoyingly, it succeeds in almost all of our projects, but there are 2 or 3 where the clone/fetch fails quite regularly (one fails 100% of the time).

0
Comment actions Permalink

Matt, could you please check does your server ask for a password when you run 'ssh <you server here>' on an agent machine?

0
Comment actions Permalink

This is the output when run from a buildagent commandline:

C:\TeamCityBuildTools\git\bin>ssh -i C:\TeamCityBuildTools\git\.ssh\id_rsa git@gitorious

Welcome, teamcity. Use git to push/pull your repositories
Connection to gitorious closed.

(The "git" user is what is set in the VCS root as the Username in the Authorization section)

0
Comment actions Permalink

Do you run an agent from the user 'git' ? If not, it will use ssh keys from its $home\.ssh folder, not from 'C:\TeamCityBuildTools\git\.ssh'.

0
Comment actions Permalink

Yes, keys were previously copied to the home folder for the user that the buildagent runs as.

Like I said before, the clones by the git plugin succeed intermittently (maybe 60% of the time), which leads me to believe that its not a permissions/key related issue, but something to do with timeouts or something else. Is there some kind of debug log we can look at for the git plugin?

0
Comment actions Permalink

To enable vcs debug log on agent change a priority of the category 'jetbrains.buildServer.VCS' from 'INFO' to 'DEBUG' in the $BuildAgentHome$/conf/teamcity-agent-log4j.xml and restart an agent.
As I understand your git server is on linux, could you please also enable debug of sshd on the server? To do that change LogLevel in the '/etc/ssh/sshd_config' to 'DEBUG3', sshd will write logs to /var/log/auth.log.

0
Comment actions Permalink

Dmitry, please send me an email to the address in my profile, and I'll send you the logs.

Thanks

0
Comment actions Permalink

Matt, I do not see an email in your profile, you can sent logs to the teamcity-feedback@jetbrains.com with the link to this thread or create an issue in http://youtrack.jetbrains.net/, attach your logs and make them visible to teamcity-developers only.

0

Please sign in to leave a comment.