Need Help - Git VCS is always timing out. (never used to :( )

Hi all,

about a week or so ago, all 7 of our builds now fail because of a timeout error. We've been using TC for about a year now without any problems at all.

Also, we have NOT made any changes, prior to the problem :(

At first, we started to 'TEST CONNECTION', which always started to return 'SUCCESS'. Then we tried to run a build and ... nope! Timeout error message :(

Now the test connectoin more or less always timesout.

So, we updated from 8.0.6 to 8.1.a. That worked fine, all 7 builds came across etc. Same thing :( (but the error message was a bit nicer, now).

Now we're using CHROME on this TC server. So opening up another tab, we goto Blamo! it pops up asap.

We're -not- using a proxy or anything fancy pants.

We have no idea how to debug this issue. We've rebooted the server. Still the same old problem :(

Is there -anything- we can do, to fix this? I couldn't find any 'timeout time' value in any of the settings. We're using the stock standard git VCS plugin that comes with TC.

This is really hurting us, cause we do commits all day long and our testing team is now .. well .. bored :( cause we can't deploy to dev!

yeah - 1st world probs i know .. but this is our last hope, Obi-wan. Someone has to be able to hopefully shed some light on something?

(I've got a weird feeling it MIGHT be a bad error message .. even though this is a GIT issue).

Details; TC v 8.01a
Repo: Private repo (soz). Using HTTPS + Username/Password for auth.

Comment actions Permalink

Do you host TeamCity server in Azure?

Comment actions Permalink

I am experiencing this sporadic issue as well. The error message is:

Failed for the root '' #33: List remote refs failed: Connection time out:   

It's been happening for quite some time. I am currently using TeamCity 8.1 and one of  the latest snapshot version of the Git plugin (SNAPSHOT-20140220022711). I have it hosted on an Azure VM (Medium Instance) which is running Windows Server 2012R2. The database is SQL Server 2012 Express installed locally on that server. The 3 build agents we use also run as 3 separate Azure VMs (Small Instances) and also run on Windows Server 2012R2.

It is very frustrating because our builds will sometimes fail with the message "Unable to collect changes." We end up having to rerun the build which then is successful. Attempts to reboot the server or restart the service do not work.


I do have IIS setup as a reverse proxy in front of TeamCity since I have other websites that are hosted under port 80 and 443. No other proxy is set up. "Test Connection" always works. I haven't tried changing the "Checking interval" for the VCS; it is set at the global default setting of 60 seconds. TeamCity version is 8.1 (build 29879).

Comment actions Permalink

Yep. We're on Azure.

OK - I've got some GOOD NEWS on this issue.

This issue is -NOT- with TeamCity at all. The issue is with AZURE. AND MICROSOFT -DO- KNOW ABOUT THIS AND ARE WORKING ON A FIX.

My TC on Azure when i try and use the git plugin with (and this is crucial) HTTPS, fails.

Goto the command line and try this:
C:\git\bin\git.exe clone

you should now get an error.

Ok, so the issue was confirmed to lie with GIT, not TC. -AND- it's only when using GIT on AZURE and against an HTTPS endpoint.

Speaking with GitHub's support they have confirmed this issue has been raised a number of times, recently. They then enacted a "deep-dive" with MS who have confirmed it's on their end (AZURE) and they have a ticket open, to fix it.

I have the details - i'll ask permission to repo the tech issue, in here. Got permission, so here's the details:

What we found was this:

* Azure's NAT implementation sometimes half-closes sockets, leaving our end with a CLOSING connection hanging out for a while

* Azure's NAT implementation re-uses ports when performing connections in quick succession

* git's --recursive clone implementation doesn't support HTTP Keepalive, which means it require's several HTTP connections to fetch the repositories and submodules

If one of the several HTTP connections that git is making to GitHub triggers the first problem and doesn't completely close, the next connection one will timeout since it's trying to use the same source port and our end thinks it's still in use.

Each of these situations alone wouldn't be enough to cause a problem, but the combination is what's making this a horrible experience.

Until Azure Networking gets a fix, then the workaround is to use SSH keys.

I can confirm that SSH KEYS are working on git + Azure and git + Azure + TC 8.1a.

Comment actions Permalink

Thank you for update and workaround! Hopefully this will help many other users who experience similar problems.
AFAIK, the problem with connectivity from Azure to github is in discussion for two weeks already. I really hope Microsoft and Github will find a resolution soon.


Please sign in to leave a comment.