Why does our TeamCity builds slow down time on VMWare?

Hi,

We are having strange problems running a TeamCity server on a VMWare hosted Windows 2008 server.

We use TeamCity to build a couple of Visual Studio 2010 solutions. On the develope machines the solutions take about a minute or so to build. On the build server, sometimes, the whole build process, including version control checkout, unit testing, making deployment packages, etc.,  takes a couple of minutes.
However very often we experience strange performance problems. In fact it has become rare that we do not experience these problems.
The symptoms are that all the build steps take up to 20 minutes and the other services running on the machine are almost unresponsive.

The odd thing that drives us crazy is that build log seams perfectly normal, even the timing. The last entries in the log have timestamps similar to when the build, occationally, runs at normal speed. When a build perform poorly another strange things happen: The system clock loses about 10-15 minutes.
So everything looks normal in logs and results except that it's like the build is made in slow-motion. For example if the build log tail is watched it ticks in with new lines occationally (very slowly), but the timestamps on the lines are only seconds apart even if the lines sometimes tick in minutes apart.

I have read blogs about people having problems with VMWare and TeamCity in a scenario where VMWare apparently also ran a tomcat which gave some conflicts, but I don't think that is the case here. There are only two tomcats running, one for TeamCity and one for YouTrack, and the problem existed before installing YouTrack.

The server configuration is a relatively new PC with plenty of ram and 12 cores. It is running VMWare with a couple of servers running. Of these only the build server does performance demanding tasks. TeamCity is 6.5.5 with the build agent running on the same machine, but the backend database (MS SQL) on another.
Executing the build steps manually on the build server runs fast and smooth. We have observed no other process or task slowing down the machine like this.

Any suggestions?

Søren

Please sign in to leave a comment.