high CPU usage an unresponsive web console
I have recently upgraded to 4.0.2 and have started to experience very high cpu usage on the server java process (99.9% almost continuously) which is experienced as a completely unresponsive web interface. The agent appear to continue to run depsite this but I am having to stop teamcity, kill the process and restart every two days or so. The java process then sits happily at about 20%.
I am at a loss to explain the cause of this issue and do not really know where to begin to try and diagnose the root cause. Has anyone else experienced this?
System details:
4.0.2 Enterprise edition
running on a Linux box 2.6.18-92.1.10.el5
mysql db on same box (at the moment)
agents all on seperate boxes
53 build configs in 13 projects
one svn vcs root
Thanks in advance
Please sign in to leave a comment.
Please install this plugin: http://www.jetbrains.net/confluence/display/TW/Server+Profiling
and try to capture CPU snapshot and upload it to our FTP server: ftp://ftp.intellij.net/.uploads
If CPU snapshot cannot be taken because of high load, try to obtain several thread dumps:
http://www.jetbrains.net/confluence/display/TCD4/Reporting+Issues#ReportingIssues-HangsandThreadDumps
you can send thread dumps to teamcity-feedback [at] jetbrains.com
The instructions seem to conflict a bit.
on the link you gave it says:
install the plugin (instructions)
The instructions on that link say
stop TeamCity
copy zip to plugins dir
start TeamCity
All ok so far. I then return to the server profiling page and read the rest of the points.
point 1 is fine
point 2 has already been done as above (but there does seem to be a mistake in it)
point 3 does not have enough info in it. What is the path to bin/<your platfom> I could not find it
point 4 was not reached
The teamcity server is back up and I can see the profilier options in the admin screen but as I have only completed half the install steps I am not prepared to switch it on yet.
any advice on this please. Thanks
Please take a look at artifacts page of the build:
http://teamcity.jetbrains.com/viewLog.html?buildId=18042&tab=artifacts&buildTypeId=bt120
There you will find bin directory. You should take native part (.so) corresponding to your platform.
oh yes, there it is! duh.
Whilst investigating I noticed a few heap sapce errors in the logs. I have increased the xmx and xms according to reccomendation and I'll see how that works before going further.
What -Xmx did you use? It is worth to add -XX:+HeapDumpOnOutOfMemoryError argument to JVM. In this case heap dump will be created automatically if out of memory occurs. You can then upload heap dumps to our ftp server: ftp://ftp.intellij.net/.uploads and send us notification to teamcity-feedback [at] jetbrains.com
In the teamcity-server.sh file I was still using the default server mem opts (eval). I have commented out that line and uncommented the recommended options a few lines below that keeping xmx at 512m and setting xms to the same but increasing perm gen space to 128m.
Here are the details of my setup for you to judge if I should increase anything further.
server cpu/ram/disk space/os/file system type:
server: Linux 2.6.18-92.1.10.el5 x86_64 GNU/Linux
8 CPU 2 GHz; 3 GB RAM; file-system-type: ext3
mysql db
size and file count in .Buildserver/system/artifacts:
1.1G in 1372 files in 348
(I have noticed that some of these artifacts are very old and no longer required. I have set the cleanup policy, but am I free to simply delete these old artifacts?)
number of projects/configurations:
12 / 52
number of source roots & which version control systems do you use:
1 VCS root using subversion 1.5.5
number of agents:
1 on teamcity server and 2 others running on seperate boxes
For your installation 512Mb of memory should be enough. But 64-bit server usually requires more memory for Java process than in 32 bit mode. Anyway it would be great if you will be able provide a memory snapshot if OOM occurs again.