We faced a really serious/strange issue with Teamcity today. One of our build agents ran out of disk space and it started throwing this exception
System.Exception: The cache file C:\Users\tfsbuild\AppData\Local\Microsoft\Team Foundation\4.0\Cache\VersionControl.config is not valid and cannot be loaded.
Please correct or delete the file.
---> Microsoft.TeamFoundation.VersionControl.Client.VersionControlException: The cache file C:\Users\tfsbuild\AppData\Local\Microsoft\Team Foundation\4.0\Cache\VersionControl.config is not valid and cannot be loaded.
We freed up disk space on the build agent but the problem, won’t go away, so we deleted the cache on the build agent and restarted both the build agent and teamcity server. After we restarted TeamCity server, it on its own scheduled around 1300 builds. That was not that big of deal as we removed everything from queue.
The serious issue was that any builds that we tried to run later were not checking out the latest code from TFS. The builds were tied to revisions in TFS that were checked in months ago, some even from last year i.e. the latest revision in TFS according to TeamCity was incorrect. We found this out by luck as we have recently changed our build scripts and we saw that the teamcity was passing in new-variables to scripts but the scripts failed to accept the variables. Had we not changed the build scripts, we might have shipped months old code.
However, we found that checking in something to TFS, made teamcity get the correct latest revision from TFS. However we had to do dummy checkins for reach VCS root.
Can you elaborate what could have gone wrong on our machines, so that we know next what to do if this happens next time.