Teamcity hanging checking for changes

Occasionally (I haven't been able to track down when), my teamcity server seems to hang...everything just sits in the build queue, and nothing gets processed.  The only thing I can see is if I go to "Administration | Diagnostics | VCS Status", it will have one of my VCS roots being checked - and it will be pending for thousands of seconds (as long as the queue has been hung).

I am not able to see anything (errors or anything) in the teamcity logs....increasing the log setting to use the vcs-debugging preset doesn't show anything new either.  It just shows that it starts to check a root, and then it never comes back.

I'm suspecting some sort of network issue between my build server and my VCS server...however, how can I clear out this "hung" connection?  The only way I can flush it through is to shut down the entire teamcity server and restart it (that seems to work, usually).

Some details of my environment:
- TeamCity version 9.0 - running on Ubuntu 14.04 (though I had been having this same issue with TeamCity version 8.1.x as well)
- Connecting via ssh to a git repository.  The git repository is hosted by an Atlassian Stash server, on the same internal network as the TeamCity server
- The root that hangs seems to be different every time.  All the repositories are very small (they have very small commit histories, and not many files)
- Restarting the git server doesn't seem to unhang the vcs check in TeamCity...only restarting the TeamCity server will flush it through

Again - I am 99% sure this is a networking issue on my end...I'm basically wondering if there is a way for me to force a VCS root to recheck (that is, to tell TeamCity to "give up") or even to lower the timeout that TeamCity will wait for "Checking for Changes" to a given VCS root.

I am able to gather additional log/thread information if needed.

2 comments
Comment actions Permalink

Hi Nathan,

What type of checkout do you use: agent or server-side? Please attach teamcity-server.log and teamcity-vcs.log files. Also please collect 5-10 server/agent thread dumps (depending on checkout mode) while the issue is reproduced.

0
Comment actions Permalink

I was doing server-side checkout.  However, I switched my repositories to be http instead of ssh, and I haven't seen the issue since.

If I see it again, I will get the needed logs.  Thank you.

0

Please sign in to leave a comment.