Git hanging

We're running TeamCity 7.0.4 on Windows 2k8 R2 x64 and having some issues with Git.  We run into an issue where git is hanging and eventually timing out.  I don't think increasing the timeout would help since it's not doing anything and it shouldn't take 10 minutes.  If I look at process explorer, I do see an ssh.exe just hanging around trying to pull that repository.  One theory I had is the repo is getting corrupted.  The reason is that restarting the web server doesn't even seem to fix it.  It just comes up and hangs again.  But if I go to the git cache directory and blow it away, it'll do a full clone and continue on for a few days until this happens again.

We do have a few builds based off this repo.  They use different branches and therefore don't share a VCS root (we have separate VCS roots for all of our configurations... is there a better way with Git?).  But I noticed that TeamCity must be smart enough to only use a single local repo.  But is it possible it's trying to update that same directory for each different configuration and corrupting it?

Thanks,
tim

8 comments
Comment actions Permalink

Hi Tim,

we synchronize write access to the local repository, so I don't think hanging caused by concurrent updates. Usually this behaviour means that repository contains some big binary objects. Please provide teamcity-vcs.log for the time TeamCity hangs and contents of 'monitoring' folder under bare repository clone on the server (you can find which directory TeamCity uses from .BuildServer/system/caches/git/map).

0
Comment actions Permalink

I've enabled debug-vcs logging and I'll post logs the next time it happens.  We really need to get this fixed, it's absolutely killing us.

0
Comment actions Permalink

Ok, it just happened with a repo of ours called Varasto.git, master branch.  You can start to see trouble on teamcity-vcs.log, line 806.  I've also attached 2 monitoring logs from that repo.  I should point out that we're running the latest version of msysgit with Cygwin's SSH (mainly because we were having trouble with the TeamCity service account and SSH keys and we've seen Cygwin's SSH client be more trustworthy).



Attachment(s):
2012-08-09_14.19.39.zip
2012-08-09_14.09.37.zip
teamcity-vcs.log.zip
0
Comment actions Permalink

One other thing odd thing we've noticed.  If I pull up Process Explorer in tree mode (so you can see child processes), the ones that hang seem to be under a java.exe (under tomcat7.exe).  Most of the others are just right under tomcat7.exe with no java.exe parent process in between.  I don't know if that means anything.

0
Comment actions Permalink

And FYI, once again with this build, no amount of TC service restarts fixed the problem.  But as soon as I deleted the local repo (under system/caches/git), it was finally able to finish.

0
Comment actions Permalink

I didn't realize that TeamCity could do Git and SSH internally.  In an attempt to alleviate the issue, I upgraded to TC 7.1 and removed the GIT_SSH and TEAMCITY_GIT_PATH env variables so now it's not shelling out to git.exe or ssh.exe.  It seems to be working now but I'll keep an eye on it.

0
Comment actions Permalink

Yes, you don't need native git on TeamCity server, but you need it on the agent if you are using agent-side checkout. Logs you provided show that fetch is inside ssh handshake. If the error will ever occur again please check if your git server logs contain anything on why handshake hangs. Maybe you reached some ssh connection limit?

0
Comment actions Permalink

Will do.  I'm sure it's something to do with SSH.  In my opinion, Git/SSH support in general on Windows is just awful.  Hopefully your internal Git/SSH client will do a better job.  If not, we may consider moving just the web server portion to a Linux box.

0

Please sign in to leave a comment.