We've been having some trouble with our teamcity server recently. As of late when doing a clean checkout the agents will hang for 3-5 hours with the following in the build log:
[13:41:42]Skip checking for changes - changes are already collected
[13:41:46][Swabra] Checkout directory contains newly created, modified or deleted files. Need a clean checkout directory snapshot - forcing clean checkout
[13:41:46]Publishing internal artifacts
[13:41:46][Publishing internal artifacts] Sending build.start.properties.gz file
[13:41:46]Clearing temporary directory: C:\BuildAgent\temp\buildTmp
[13:41:46]Checkout directory: C:\BuildAgent\work\517214fa61cfd7bf
[13:41:46]Updating sources: server side checkout
[13:41:46][Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
As well when you attempt to test the connections to our VCS roots that hangs as well, however it is still sucessful in grabbing the latest changes. We currently use a perforce source control system and are using a dated 7.1.4 teamcity.
This is recent, as in the last 2-3 months.
I've attached a couple of thread dumps that were taken while the agents were "hung" checking out. I tried to get agent thread dumps but got an error message. Any ideas? Our VCS server is aged and not in the healthiest condition, but shouldn't be causing this especially since the server is setup for server side checkout.