Hi!
We're getting the warning
"TeamCity server memory usage for PS Old Gen pool exceeded 92% of 800 MB maximum available. 847 MB used of 1 GB total heap available"
and performance is sluggish - rebooting helps briefly, but then it goes bad again.
We have just upgraded from 9.0 to 9.1, but it didn't help. We're running on Windows Server and MS SQL Server using 32 bit java with the recommended memory settings.
We have two build agents (one local, one remote) and are building.net solutions using Perforce as source control. We have had some issues with looping builds - hopefully these are now resolved.
How do I troubleshoot this issue?
Here's a snapshot of the memory graph:
I have pasted in the usage statistics below, but I'm not sure what to look for?
General
| Meta-Runners |
5 |
| Connected agents (all) |
2 |
| Connected agents (authorized only) |
2 (100%) |
| Agent pools |
1 |
| Build configurations |
540 |
| Active build configurations |
230 (42%) |
| Active build configurations with several VCS roots |
0 (0%) |
| Snapshot dependencies |
425 |
| Artifact dependencies |
390 |
| Projects |
489 |
| Archived projects |
283 (57%) |
| User groups |
1 |
| Users |
12 |
| VCS roots |
352 |
NuGet Feed Usage
| Feed Requests Count per Day |
0 |
| Packages Count |
2 |
| Different Package Ids Count |
2 |
Server Configuration
| Server ID |
30c87ca4-87b0-48c6-bdc6-103096bd2aaa |
| Platform |
Windows Server 2012 6.2 x86 |
| Login modules |
Microsoft Windows domain |
| HTTP authentication schemes |
Basic HTTP, NTLM HTTP |
| Database version |
Microsoft SQL Server 10.50 |
| JDBC driver version |
Microsoft JDBC Driver 4.1 for SQL Server 4.1 |
| Java version |
1.8.0_45 |
| Java runtime version |
1.8.0_45-b15 |
| Servlet container |
Apache Tomcat/7.0.59 |
| Maximum available memory |
1185Mb |
| License type |
Enterprise |
| License mode |
Enterprise |
| Agent licenses |
3 |
| Server version |
9.1 |
| Server build |
36973 |
| Current uptime |
1h:13m |
Server Load
|
Day |
Week |
Month |
| Build count |
3999 |
27671 |
86986 |
| Personal build count |
0 |
0 |
0 |
| Average build waiting in queue time |
4m:32s |
55s |
34s |
| Average build duration |
37s |
32s |
27s |
| Maximum test count per build |
44 |
44 |
44 |
| Web users |
1 (8%) |
2 (16%) |
5 (41%) |
| IDE users |
0 (0%) |
0 (0%) |
0 (0%) |
| Web only users |
1 (8%) |
2 (16%) |
5 (41%) |
| IDE only users |
0 (0%) |
0 (0%) |
0 (0%) |
| VCS changes |
20 |
46 |
366 |
Runners
| Command Line |
39 (16%) |
| Duplicates finder (.NET) |
8 (3%) |
| FxCop |
8 (3%) |
| Inspections (.NET) |
5 (2%) |
| Move nuget.exe to temp dir and replace with link |
41 (17%) |
| MSBuild |
161 (70%) |
| NuGet Installer |
54 (23%) |
| Patch hintpaths for release |
126 (54%) |
| Powershell |
14 (6%) |
| Visual Studio (sln) |
10 (4%) |
| Visual Studio Tests |
4 (1%) |
Build Features
| AssemblyInfo patcher |
159 (69%) |
| Build files cleaner (Swabra) |
15 (6%) |
Build Failure Conditions
| Fail build on metric change |
4 (1%) |
Web Browsers
|
Day |
Week |
Month |
| Chrome |
1 (100%) |
2 (100%) |
4 (80%) |
| Firefox |
0 (0%) |
0 (0%) |
1 (20%) |
| Internet Explorer 10 |
0 (0%) |
0 (0%) |
1 (20%) |
Authentication Modules
|
Day |
Week |
Month |
| Microsoft Windows domain |
0 (0%) |
0 (0%) |
3 (25%) |
| NTLM HTTP |
0 (0%) |
1 (8%) |
3 (25%) |
VCS Features
| Git VCS roots with submodules support enabled |
0 (0%) |
| Mercurial VCS roots with subrepo support enabled |
0 (0%) |
| Subversion VCS roots with externals support enabled |
0 (0%) |
Triggers
| Finish Build Trigger |
9 (3%) |
| Schedule Trigger |
6 (2%) |
| VCS Trigger |
176 (76%) |
Notifiers
| Windows Tray Notifier |
1 (8%) |
Investigation / Mute
|
Day |
Week |
Month |
| Build configuration investigations |
0 |
0 |
0 |
| Test investigations |
0 |
0 |
0 |
| Test mutes |
0 |
0 |
0 |
IDE Features
|
Day |
Week |
Month |
| Remote Debug |
0 |
0 |
0 |
| Remote Debug (sessions) |
0 |
0 |
0 |
| Test Status (with successful) |
0 |
0 |
0 |
| Test Status (without successful) |
0 |
0 |
0 |
Coverage Engines
| JetBrains dotCover |
2 (0%) |
Web Pages Usage
|
Week |
Month |
| /agentDetails.html |
2 (100%) |
2 (40%) |
| /agentDetails.html?tab=agentCompatibleConfigurations |
2 (100%) |
2 (40%) |
| /agentDetails.html?tab=agentSummary |
1 (50%) |
1 (20%) |
| /allProjects.html |
1 (50%) |
2 (40%) |
| /changesPopup.html |
1 (50%) |
3 (60%) |
| /diffView.html |
0 (0%) |
1 (20%) |
| /filesPopup.html |
1 (50%) |
2 (40%) |
| /overview.html |
2 (100%) |
4 (80%) |
| /project.html |
1 (50%) |
2 (40%) |
| /project.html?tab=projectChangeLog |
1 (50%) |
1 (20%) |
| /project.html?tab=projectOverview |
1 (50%) |
3 (60%) |
| /promoDetailsPopup.html |
1 (50%) |
1 (20%) |
| /remoteAccess/rdp.html |
1 (50%) |
1 (20%) |
| /runCustomBuild.html |
1 (50%) |
3 (60%) |
| /systemProblems.html |
1 (50%) |
3 (60%) |
| /viewArtifactsPopup.html |
1 (50%) |
4 (80%) |
| /viewLog.html |
1 (50%) |
2 (40%) |
| /viewLog.html?tab=buildLog |
1 (50%) |
3 (60%) |
| /viewLog.html?tab=buildResultsDiv |
1 (50%) |
2 (40%) |
| /viewLog.html?tab=Duplicator |
0 (0%) |
1 (20%) |
| /viewLog.html?tab=Inspection |
0 (0%) |
1 (20%) |
| /viewModification.html?tab=vcsModificationFiles |
1 (50%) |
1 (20%) |
| /viewType.html |
1 (50%) |
3 (60%) |
| /viewType.html?tab=buildTypeChangeLog |
1 (50%) |
1 (20%) |
Hi Niels,
Please follow the recommendations from the guideline: https://confluence.jetbrains.com/display/TCD9/Reporting+Issues#ReportingIssues-OutOfMemoryProblems.
If memory settings tuning does not help, then please attach memory dump.
We are running with -Xmx1200m -XX:MaxPermSize=270m
After running a cleanup, the issue seems to have gone away - memory consumption is now down:
Total heap: 732.95 MB (69% of maximum available 1.04 GB)
Data: 547.52 MB (68% of maximum available 800 MB)
I assume that the cleanup has flushed some of the old build info from the caches.
I will not do more about for now but watch for further problems.
Thanks!
Glad it works! It is recommended to configure clean-up rules to remove obsolete builds and their artifacts, purge unnecessary data from database and caches in order to free disk space, remove builds from the TeamCity UI and reduce the TeamCity workload. And to run clean-up on regular basis.
Unfortunately, the server is acting up again.
I have uploaded a memory dump and some thread dumps to ftp://ftp.intellij.net/.uploads
- the filename is teamcity_memory_dumps_bruel_og_kjaer.zip
It looks like your server produces fair amount of builds every day. I'd recommend configuring stricter cleanup rules in this case or switch to 64bit JVM and give TeamCity server more memory.
Also in upcoming 9.1.1 bugfix release we added some related memory usage improvements It should be published on the next week.