Occasionally (I haven't been able to track down when), my teamcity server seems to hang...everything just sits in the build queue, and nothing gets processed. The only thing I can see is if I go to "Administration | Diagnostics | VCS Status", it will have one of my VCS roots being checked - and it will be pending for thousands of seconds (as long as the queue has been hung).
I am not able to see anything (errors or anything) in the teamcity logs....increasing the log setting to use the vcs-debugging preset doesn't show anything new either. It just shows that it starts to check a root, and then it never comes back.
I'm suspecting some sort of network issue between my build server and my VCS server...however, how can I clear out this "hung" connection? The only way I can flush it through is to shut down the entire teamcity server and restart it (that seems to work, usually).
Some details of my environment:
- TeamCity version 9.0 - running on Ubuntu 14.04 (though I had been having this same issue with TeamCity version 8.1.x as well)
- Connecting via ssh to a git repository. The git repository is hosted by an Atlassian Stash server, on the same internal network as the TeamCity server
- The root that hangs seems to be different every time. All the repositories are very small (they have very small commit histories, and not many files)
- Restarting the git server doesn't seem to unhang the vcs check in TeamCity...only restarting the TeamCity server will flush it through
Again - I am 99% sure this is a networking issue on my end...I'm basically wondering if there is a way for me to force a VCS root to recheck (that is, to tell TeamCity to "give up") or even to lower the timeout that TeamCity will wait for "Checking for Changes" to a given VCS root.
I am able to gather additional log/thread information if needed.