Stuck in Updating sources

Generally our projects (with the exception of one) run with little to no issues.
A little background - all of our projects except one (1) are under 900 MB compiled. One project is ~2.8 GB compiled.
We strictly use TFS 2008 for our source control.


This our general problem resolution for Team City:
There are times when the projects will seem to be stuck in "Updating sources: Building and caching clean patch for VCS root: .....". Our usual method of dealing with this is give it about 15 minutes if it does not show any change - we stop the build. Then we try running the build again, this will work sometimes. If this second build does not work then we run a custom build with "clean all files in the checkout directory before the build", this will sometimes work. After this if the build does not work we will try another regular build, this will work sometimes.

We have now exhausted our patience with a build and spent almost an hour to push a build.
Now we do what is probably something that we should not do, however it does seem to resolve the issue:

  1. Login to the Team City Server
  2. Stop the TeamCity service
  3. Stop the TCBuildAgent service
  4. Navigate to \TeamCity\ProgramData\system
  5. Delete the caches folder
  6. Restart the TeamCity service
  7. Restart the TCBuildAgent service


After performing these steps the build will run.

As I write this, a project has gone into a state of hung Updating Sources.... - luckily eariler I had turned all debug logging on.

I grabbed a download of all the log files but for right now I am attaching the team city server log file and thread dump. I did also grab a memory dump but the file is 46 MB.

Any help would be appreciated. If there is an



Attachment(s):
2015-07-30_15.04.28-manual.txt.zip
teamcity-server.log.zip
2 comments
Comment actions Permalink

Hi,

Sorry for delay.
Could you please fill the issue in our tracker? Please use this link. Also it would be great if you collect several thread dumps while server is in hanging state.

0
Comment actions Permalink

I have created the issue as directed and have been adding to it.
https://youtrack.jetbrains.com/issue/TW-42108

0

Please sign in to leave a comment.