Huge Performance Problems with TeamCity 3.0.1

I've updated my TeamCity 2 Installation with TeamCity 3.0.1. The first
two weeks everything worked fine. The build performance even increased.
Suddenly after the first two weeks the performance dropped radically.
Unit Tests that before took 1 minute, now take 14 minutes. Compilation
takes now 6 minutes instead of 2 minutes.

When I execute the msbuild and nant files manuelly on the the server,
everything is fine. When executing via TeamCity it is very slow together
with a very low CPU usage. Is a TeamCity component waiting for some events?

What can I do to fix that problem or to get more info about it?

I've uploaded the Log files (buildAgent_logs.zip, teamcity_logs.zip) to
ftp://ftp.intellij.net/.uploads

My Configuration:
- Win 2003 Server SP2 SE 32 bit
- MySQL 5.0.37 via TCP

Thanks,
Stephan

4 comments

Stephan Brenner wrote:

When I execute the msbuild and nant files manuelly on the the server,
everything is fine. When executing via TeamCity it is very slow together
with a very low CPU usage. Is a TeamCity component waiting for some events?


Do you have a lot of disk activity? A lot of memory in use? (Process
Explorer is great for this). Maybe something's runaway with all the
memory and you're getting page file thrashing ... ?

0

Chris Morris wrote:

Stephan Brenner wrote:

>> When I execute the msbuild and nant files manuelly on the the server,
>> everything is fine. When executing via TeamCity it is very slow
>> together with a very low CPU usage. Is a TeamCity component waiting
>> for some events?


Do you have a lot of disk activity? A lot of memory in use? (Process
Explorer is great for this). Maybe something's runaway with all the
memory and you're getting page file thrashing ... ?


Hi Chris,

I've checked that with process explorer. During running Unit Tests (that
normally take 1 minute, but now 14 minutes), there is nearly no CPU and
IO activity (see attachment). Also the memory usage is rather low
(Agent, Java and Runner take together approximately 50 MB).

Cheers,
Stephan



Attachment(s):
process explorer.png
0

Stephan Brenner wrote:

Chris Morris wrote:

>> Stephan Brenner wrote:
>>> When I execute the msbuild and nant files manuelly on the the server,
>>> everything is fine. When executing via TeamCity it is very slow
>>> together with a very low CPU usage. Is a TeamCity component waiting
>>> for some events?
>>
>> Do you have a lot of disk activity? A lot of memory in use? (Process
>> Explorer is great for this). Maybe something's runaway with all the
>> memory and you're getting page file thrashing ... ?


Hi Chris,

I've checked that with process explorer. During running Unit Tests (that
normally take 1 minute, but now 14 minutes), there is nearly no CPU and
IO activity (see attachment). Also the memory usage is rather low
(Agent, Java and Runner take together approximately 50 MB).

Cheers,
Stephan

------------------------------------------------------------------------


I found out, that when I disconnect the server from the network,
everything is fast again. Agent and WebServer on the same Server and are
using 127.0.0.1 to connect to each other, but nonetheless somehow the
Network Connection gets used.

0

Stephan,

Thank you for the investigation. I've created an issue to address the problem: http://www.jetbrains.net/jira/browse/TW-4375

We will look into it as soon as possible.

Please watch it so we can communicate if further information is needed.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Please sign in to leave a comment.