high CPU usage an unresponsive web console

I have recently upgraded to 4.0.2 and have started to experience very high cpu usage on the server java process (99.9% almost continuously) which is experienced as a completely unresponsive web interface. The agent appear to continue to run depsite this but I am having to stop teamcity, kill the process and restart every two days or so. The java process then sits happily at about 20%.
I am at a loss to explain the cause of this issue and do not really know where to begin to try and diagnose the root cause. Has anyone else experienced this?

System details:
4.0.2 Enterprise edition
running on a Linux box 2.6.18-92.1.10.el5
mysql db on same box (at the moment)
agents all on seperate boxes
53 build configs in 13 projects
one svn vcs root

Thanks in advance

7 comments
Comment actions Permalink

Please install this plugin: http://www.jetbrains.net/confluence/display/TW/Server+Profiling
and try to capture CPU snapshot and upload it to our FTP server: ftp://ftp.intellij.net/.uploads

If CPU snapshot cannot be taken because of high load, try to obtain several thread dumps:
http://www.jetbrains.net/confluence/display/TCD4/Reporting+Issues#ReportingIssues-HangsandThreadDumps

you can send thread dumps to teamcity-feedback [at] jetbrains.com

0
Comment actions Permalink

The instructions seem to conflict a bit.

on the link you gave it says:
install the plugin (instructions)

The instructions on that link say
stop TeamCity
copy zip to plugins dir
start TeamCity

All ok so far. I then return to the server profiling page and read the rest of the points.
point 1 is fine
point 2 has already been done as above (but there does seem to be a mistake in it)
point 3 does not have enough info in it. What is the path to bin/<your platfom>  I could not find it
point 4 was not reached

The teamcity server is back up and I can see the profilier options in the admin screen but as I have only completed half the install steps I am not prepared to switch it on yet.

any advice on this please. Thanks

0
Comment actions Permalink

Please take a look at artifacts page of the build:
http://teamcity.jetbrains.com/viewLog.html?buildId=18042&tab=artifacts&buildTypeId=bt120

There you will find bin directory. You should take native part (.so) corresponding to your platform.

0
Comment actions Permalink

oh yes, there it is! duh.

Whilst investigating I noticed a few heap sapce errors in the logs. I have increased the xmx and xms according to reccomendation and I'll see how that works before going further.

0
Comment actions Permalink

What -Xmx did you use? It is worth to add  -XX:+HeapDumpOnOutOfMemoryError argument to JVM. In this case heap dump will be created automatically if out of memory occurs. You can then upload heap dumps to our ftp server: ftp://ftp.intellij.net/.uploads and send us notification to teamcity-feedback [at] jetbrains.com

0
Comment actions Permalink

In the teamcity-server.sh file I was still using the default server mem opts (eval). I have commented out that line and uncommented the recommended options a few lines below that keeping xmx at 512m and setting xms to the same but increasing perm gen space to 128m.

Here are the details of my setup for you to judge if I should increase anything further.

server cpu/ram/disk space/os/file system type:
server: Linux 2.6.18-92.1.10.el5 x86_64 GNU/Linux
8 CPU 2 GHz; 3 GB RAM; file-system-type: ext3
mysql db

size and file count in .Buildserver/system/artifacts:
1.1G in 1372 files in 348
(I have noticed that some of these artifacts are very old and no longer required. I have set the cleanup policy, but am I free to simply delete these old artifacts?)

number of projects/configurations:
12 / 52

number of source roots &  which version control systems do you use:
1 VCS root using subversion 1.5.5

number of agents:
1 on teamcity server and 2 others running on seperate boxes

0
Comment actions Permalink

For your installation 512Mb of memory should be enough. But 64-bit server usually requires more memory for Java process than in 32 bit mode. Anyway it would be great if you will be able provide a memory snapshot if OOM occurs again.

0

Please sign in to leave a comment.