TeamCity Agents Hang during a clean checkout.

We've been having some trouble with our teamcity server recently. As of late when doing a clean checkout the agents will hang for 3-5 hours with the following in the build log:

[13:41:42]Skip checking for changes - changes are already collected

[13:41:46]Swabra

[13:41:46][Swabra] Checkout directory contains newly created, modified or deleted files. Need a clean checkout directory snapshot - forcing clean checkout

[13:41:46]Publishing internal artifacts

[13:41:46][Publishing internal artifacts] Sending build.start.properties.gz file

[13:41:46]Clearing temporary directory: C:\BuildAgent\temp\buildTmp

[13:41:46]Checkout directory: C:\BuildAgent\work\517214fa61cfd7bf

[13:41:46]Updating sources: server side checkout

[13:41:46][Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist


As well when you attempt to test the connections to our VCS roots that hangs as well, however it is still sucessful in grabbing the latest changes. We currently use a perforce source control system and are using a dated 7.1.4 teamcity.

This is recent, as in the last 2-3 months.

I've attached a couple of thread dumps that were taken while the agents were "hung" checking out. I tried to get agent thread dumps but got an error message. Any ideas? Our VCS server is aged and not in the healthiest condition, but shouldn't be causing this especially since the server is setup for server side checkout.


Attachment(s):
threadDump-2014-12-08_14.55.13.txt.zip
threadDump-2014-12-08_14.55.04.txt.zip
4 comments
Comment actions Permalink

I forgot to add that I can open the P4V client on the server itself and browse and get through perforce no problem. It is just an issue within TeamCity itself.

Also, these builds will eventually pull latest from the server, however it takes around 3 hours for them to start getting.

0
Comment actions Permalink

Please upgrade to the latest TeamCity version (8.1.5 as of now) and check if the issue is still actual. If it reproduces on the latest version we will be interested in investigating the case in more detail. I do not think an investigation makes sense in earlier TeamCity versions as the issue might not be actual already and even if not, we will not be able to provide any patches/fixes to an obsolete version.

0
Comment actions Permalink

Unfortunately that is not an option for us at this point. We are trying to build a release that we cannot build because of this issue with TeamCity. We currently do not have a license to upgrade to the latest version, and doing so requires a formal process with our internal IT group that could take several weeks.

Another issue that seems to point at this is when you hit the test connection button when you go to edit a VCS root it hangs and never returns a message.

0
Comment actions Permalink

After finally getting a response from our IT group it looks like they were running a backup on the Host server that our TeamCity instance is on that was taking around 20 hours a day to complete. This was causing slow read speeds for the server. As soon as they stopped that our checkout times returned to normal.

0

Please sign in to leave a comment.