TeamCity server takes too much CPU and memory

Hi,

Sometimes it happens that TeamCity server freeze in the web interface,
impossible to load a page...

When I look on server, top command says this :

PID    USER     PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27089 www-data  25   0 1130m 374m  11m S   78 11.4   2685:50 java

In catalina.out I see that :

Cannot call setFile(): java.io.FileNotFoundException:
../logs/teamcity-server.log (No such file or directory)
  ERROR -   jetbrains.buildServer.SERVER -
java.lang.OutOfMemoryError: PermGen space
Exception in thread "Ping" java.lang.OutOfMemoryError: PermGen space
Cannot call setFile(): java.io.FileNotFoundException:
../logs/teamcity-server.log (No such file or directory)


I've seen that playing with memory parameters could help. But why is
this happening, I've been using it for monthes now, I didn't see this
before. I upgrade to 5.1 last week, but problem was present with 5.0.1.

Second point, why does it not found teamcity-server.log file ?
Where could I configure this?

Thanks,

Gérald

12 comments

Hello,

  You get java.lang.OutOfMemoryError: PermGen space error. That means, there is some memory issues with the server.
  Could you please increase JVM memory parameter for PermGen space, like  -XX:MaxPermSize=200m

  Please read more about OutOfMemoryError errors in our docs.

  Hope this helps,
  KIR

0

Le 26/04/2010 13:24, Kirill Maximov a ecrit :

Hello,

>

   You get java.lang.OutOfMemoryError: PermGen space error. That means, there is some memory issues with the server.
   Could you please increase JVM memory parameter for PermGen space, like  -XX:MaxPermSize=200m


Yep, it's done.

>

   Please read more about OutOfMemoryError errors in http://confluence.jetbrains.net/display/TCD5/Reporting+Issues#ReportingIssues-OutOfMemoryProblems.

>

   Hope this helps,


I hope too. I'll let you know if it happens again.

Gérald

0

Le 26/04/2010 13:24, Kirill Maximov a ecrit :

Hello,

>

   You get java.lang.OutOfMemoryError: PermGen space error. That means, there is some memory issues with the server.
   Could you please increase JVM memory parameter for PermGen space, like  -XX:MaxPermSize=200m

>

   Please read more about OutOfMemoryError errors in http://confluence.jetbrains.net/display/TCD5/Reporting+Issues#ReportingIssues-OutOfMemoryProblems.


It doesn't help.. Still the same issue.

I've read documentation, and set -XX:+HeadDumpOnOutOfMemoryError
parameter, I hope it will work.

Something funny, or I don't understand the documentation :

     Since TeamCity 5.1 you can dump memory snapshot right from the web UI:
go to the Administration | Server Configuration | Diagnostics tab and
click Dump Memory Snapshot.


How do I get a memory snapshot with webUI as server is taking too much
memory and process and doesn't respond? :)

Gérald

0

Could you please describe your current memory settings? Do you use 64-bit JVM or 32-bit?

As for memory dump on diagnostic page - of course you won't be able to use it if server does not respond, but still it is useful in some cases when server is given more memory than it should use, and you see the high memory usage.

0

Le 27/04/2010 19:40, Pavel Sher a ecrit :

Could you please describe your current memory settings? Do you use 64-bit JVM or 32-bit?


At first it was this :
#TEAMCITY_SERVER_MEM_OPTS="-Xmx512m -XX:MaxPermSize=150m"

Then I tried this :

  1. Options recommended for dedicated server installation (commented by

default)
#TEAMCITY_SERVER_MEM_OPTS="-Xms750m -Xmx750m -XX:MaxPermSize=200m"

Now it's this :
TEAMCITY_SERVER_MEM_OPTS="-Xmx512m -XX:MaxPermSize=200m
-XX:+HeapDumpOnOutOfMemoryError"

It's 32 bits, Ubuntu 7.10, with 3Gb RAM

As for memory dump on diagnostic page - of course you won't be able to use it if server does not respond, but still it is useful in some cases when server is given more memory than it should use, and you see the high memory usage.


Ok.

Gérald

0

Ok, once you'll have memory dump please upload it to our ftp according to instructions about out of memory problems, and send name of the file to: teamcity-feedback[at]jetbrains.com

0

Le 27/04/2010 20:07, Pavel Sher a ecrit :

Ok, once you'll have memory dump please upload it to our ftp according to instructions about out of memory problems, and send name of the file to: teamcity-feedback[at]jetbrains.com


It's happening again (it's happening everyday :( ) but I don't see any
.hprof file in bin directory... Does it dumps itself? How long is it to
dump it?

Gérald

0

Le 28/04/2010 16:24, Gérald Fauvelle a ecrit :

Le 27/04/2010 20:07, Pavel Sher a ecrit :

>>  Ok, once you'll have memory dump please upload it to our ftp according to instructions about out of memory problems, and send name of the file to: teamcity-feedback[at]jetbrains.com
>

It's happening again (it's happening everyday :( ) but I don't see any
.hprof file in bin directory... Does it dumps itself? How long is it to
dump it?


How can I know if options I give are use by TeamCityserver?

If I do a ps to see the proces, I see this as a command line used to run
TeamCity:

/usr/lib/jvm/java-6-sun/bin/java
-Djava.util.logging.config.file=/usr/local/TeamCity/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/usr/local/TeamCity/endorsed -classpath
/usr/local/TeamCity/bin/bootstrap.jar
-Dcatalina.base=/usr/local/TeamCity -Dcatalina.home=/usr/local/TeamCity
-Djava.io.tmpdir=/usr/local/TeamCity/temp
org.apache.catalina.startup.Bootstrap start


Gérald

0

.hprof file must be created in the process working directory. Did you restart server after the error? If not, try to dump memory from the diagnostic tab and upload it to our ftp.

0

Le 28/04/2010 19:00, Pavel Sher a ecrit :

hprof file must be created in the process working directory.

> Did you restart server after the error?

Yes, I did it again.

> If not, try to dump memory from the diagnostic tab and upload
> it to our ftp.

Yeah, but the problem is that the server doesn't respond at all... It
takes 100% cpu and web interface is not loaded.

Gérald

0

TEAMCITY_MEM_OPTS are only used if TeamCity is started by our own .sh files, i.e. by teamcity-server.sh or by runAll.sh
If you deploy TeamCity war to already started Tomcat container or if you started TeamCity by startup.sh, then these settings won't be passed to JVM.
In the latter case you need to use JAVA_OPTS environment variable, I guess this is the case here.

0

Le 28/04/2010 19:36, Pavel Sher a ecrit :

TEAMCITY_MEM_OPTS are only used if TeamCity is started by our own .sh files, i.e. by teamcity-server.sh or by runAll.sh
If you deploy TeamCity war to already started Tomcat container or if you started TeamCity by startup.sh, then these settings won't be passed to JVM.
In the latter case you need to use JAVA_OPTS environment variable, I guess this is the case here.
  

I use teamcity-server.sh. I put some echo $TEAMCITY_MEM_OPTS in the
script, it seems the parameters are defined.

I've restarted the server, I'll come back when it happens again.

Gérald

0

Please sign in to leave a comment.