TeamCity 2.1.1 memory leak?

Every 3-5 days we have to restart the TeamCity service (not the agent) because it stops working.
In the logfile, the following line can be found:

WARN - r.controllers.XmlRpcController - Exception occured while communicating by XML RPC: java.lang.OutOfMemoryError: Java heap space

Is this a known problem that occurs elsewhere as well?

13 comments

Tobias Kurzke wrote:

Every 3-5 days we have to restart the TeamCity service (not the agent) because it stops working.
In the logfile, the following line can be found:

WARN - r.controllers.XmlRpcController - Exception occured while communicating by XML RPC: java.lang.OutOfMemoryError: Java heap space

Is this a known problem that occurs elsewhere as well?


Our server runs for weeks w/o any problems under quite heavy load.
So we assume there are no significant memory leaks in TC.

Do you mean that OOME occur again after you restart the server?

If so, please send us server logs to investigate.


--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0

Any news on this one already?
We have to keep restarting TeamCity every week or so meanwhile...

0

Tobias Kurzke wrote:

Any news on this one already?
We have to keep restarting TeamCity every week or so meanwhile...


All symptoms that your load is too high for default embedded hsql
configuration you have. We strongly recommend you to upgrade to mysql.

If provide us with some additional information, we can give you advice
on server configuration. (see list below)

server cpu/ram/disk space/os/file system type
size and file count in .Buildserver/system/artifacts
number of projects/configurations
number of source roots & which version control systems do you use
number of agents

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0

Hi Alexey,
excuse me for responding so late. Here the answers to your questions:

1. server cpu/ram/disk space/os/file system type
server: Microsoft Windows Server 2003 (Service Pack 2),
AMD opteron Processor 275; 2,21 GHz; 3,83 GB RAM; file-type: NTFS

2. agent: The agent is in a VM - Ware - Session:
Microsoft Windows XP Professional (Version 2002), ServicePack 2, 512 MB RAM

3. size and file count in .Buildserver/system/artifacts
size: 83 MB, files: 260, folder: 73

4. number of projects/configurations:
number of projects: 11
number of configurations: 57

5. number of source roots & which version control systems do you use
We use subversion 1.4.4
we have two svn - repos
9 of our projects use svn:externals
2 of our project use a source root

6. number of agents: 1

0

Florian v.Schoenebeck wrote:

One more question - how do you start TeamCity - as windows service
(installed by our installer), using shell script or in some other way?

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0

We start TeamCity as windows service.
If we have to restart TeamCity we do it manually.

0

Florian v.Schoenebeck wrote:

We start TeamCity as windows service.
If we have to restart TeamCity we do it manually.


Can you please run

tomcat5w.exe //ES//TeamCity

and ensure that "maximum heap" size is around 300-500 Megs

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0

Thanks for this advice.
I hope I understand it in the right way:
- I started the GUI with >tomcat5w.exe //ES//TeamCity
- in the tab "Java" I set the following values:

  • initial memory pool = 300

  • maximum memory pool = 500

0

Florian v.Schoenebeck wrote:

Thanks for this advice.
I hope I understand it in the right way:
- I started the GUI with >tomcat5w.exe //ES//TeamCity
- in the tab "Java" I set the following values:

  • initial memory pool = 300

  • maximum memory pool = 500

Yes, this is OK.
To improve performance I may suggest you to switch to latest server JVM.
Just point server JVM path to something like
C:\java\jdk\jdk16u2\jre\bin\server\jvm.dll

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0

Since two weeks we did not restart the server, thus the configuration you gave us works.

Today I saw the following message in the log - file:

+ WARN - rver.serverSide.VersionChecker - Unable to check for new version: java.net.ConnectException: Connection timed out: connect
Exception in thread "Agent Distribution Monitor" java.lang.OutOfMemoryError: Java heap space+

I think these are two problems:
1. VersionChecker - I don't know how to fix this
2. OutOfMemoryError - I don't understand this exception because the agent works and I didn't have to restart him.

0

Hi Alexey,
after three weeks we got our known error:
+ WARN - r.controllers.XmlRpcController - Exception occured while communicating by XML RPC: java.lang.OutOfMemoryError: Java heap space +

Thus, in my eyes, there is a memory leak.
With the new memory settings we do not have to restart every 3-5 day, but we have to restart every 3 weeks.

Florian v.Schoenebeck

0

Florian v.Schoenebeck wrote:

Hi Alexey,
after three weeks we got our known error:
+ WARN - r.controllers.XmlRpcController - Exception occured while communicating by XML RPC: java.lang.OutOfMemoryError: Java heap space +

Thus, in my eyes, there is a memory leak.
With the new memory settings we do not have to restart every 3-5 day, but we have to restart every 3 weeks.

Florian v.Schoenebeck


If this is not acceptable I suggest you to try TC 3.0 EAP. It contains
many performance improvements and is very stable. Anyway, if you decide
to do so please do not forget make backup of database and TC
configuration. BTW, next EAP will be released to public in next 2-3 days.
--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0

Please sign in to leave a comment.