Troubleshooting memory usage

Hi!
We're getting the warning

"TeamCity server memory usage for PS Old Gen pool exceeded 92% of 800 MB maximum available. 847 MB used of 1 GB total heap available"

and performance is sluggish - rebooting helps briefly, but then it goes bad again.
We have just upgraded from 9.0 to 9.1, but it didn't help. We're running on Windows Server and MS SQL Server using 32 bit java with the recommended memory settings.

We have two build agents (one local, one remote) and are building.net solutions using Perforce as source control. We have had some issues with looping builds - hopefully these are now resolved.


How do I troubleshoot this issue?


Here's a snapshot of the memory graph:

2015-07-23 17_19_46-Diagnostics _ Troubleshooting — TeamCity.png

I have pasted in the usage statistics below, but I'm not sure what to look for?

General
Meta-Runners 5
Connected agents (all) 2
Connected agents (authorized only) 2 (100%)
Agent pools 1
Build configurations 540
Active build configurations 230 (42%)
Active build configurations with several VCS roots 0 (0%)
Snapshot dependencies 425
Artifact dependencies 390
Projects 489
Archived projects 283 (57%)
User groups 1
Users 12
VCS roots 352
NuGet Feed Usage
Feed Requests Count per Day 0
Packages Count 2
Different Package Ids Count 2
Server Configuration
Server ID 30c87ca4-87b0-48c6-bdc6-103096bd2aaa
Platform Windows Server 2012 6.2 x86
Login modules Microsoft Windows domain
HTTP authentication schemes Basic HTTP, NTLM HTTP
Database version Microsoft SQL Server 10.50
JDBC driver version Microsoft JDBC Driver 4.1 for SQL Server 4.1
Java version 1.8.0_45
Java runtime version 1.8.0_45-b15
Servlet container Apache Tomcat/7.0.59
Maximum available memory 1185Mb
License type Enterprise
License mode Enterprise
Agent licenses 3
Server version 9.1
Server build 36973
Current uptime 1h:13m
Server Load
Day Week Month
Build count 3999 27671 86986
Personal build count 0 0 0
Average build waiting in queue time 4m:32s 55s 34s
Average build duration 37s 32s 27s
Maximum test count per build 44 44 44
Web users 1 (8%) 2 (16%) 5 (41%)
IDE users 0 (0%) 0 (0%) 0 (0%)
Web only users 1 (8%) 2 (16%) 5 (41%)
IDE only users 0 (0%) 0 (0%) 0 (0%)
VCS changes 20 46 366
Runners
Command Line 39 (16%)
Duplicates finder (.NET) 8 (3%)
FxCop 8 (3%)
Inspections (.NET) 5 (2%)
Move nuget.exe to temp dir and replace with link 41 (17%)
MSBuild 161 (70%)
NuGet Installer 54 (23%)
Patch hintpaths for release 126 (54%)
Powershell 14 (6%)
Visual Studio (sln) 10 (4%)
Visual Studio Tests 4 (1%)
Build Features
AssemblyInfo patcher 159 (69%)
Build files cleaner (Swabra) 15 (6%)
Build Failure Conditions
Fail build on metric change 4 (1%)
Web Browsers
Day Week Month
Chrome 1 (100%) 2 (100%) 4 (80%)
Firefox 0 (0%) 0 (0%) 1 (20%)
Internet Explorer 10 0 (0%) 0 (0%) 1 (20%)
Authentication Modules
Day Week Month
Microsoft Windows domain 0 (0%) 0 (0%) 3 (25%)
NTLM HTTP 0 (0%) 1 (8%) 3 (25%)
VCS Root Types
Perforce 352 (100%)
VCS Features
Git VCS roots with submodules support enabled 0 (0%)
Mercurial VCS roots with subrepo support enabled 0 (0%)
Subversion VCS roots with externals support enabled 0 (0%)
Triggers
Finish Build Trigger 9 (3%)
Schedule Trigger 6 (2%)
VCS Trigger 176 (76%)
Notifiers
Windows Tray Notifier 1 (8%)
Investigation / Mute
Day Week Month
Build configuration investigations 0 0 0
Test investigations 0 0 0
Test mutes 0 0 0
IDE Features
Day Week Month
Remote Debug 0 0 0
Remote Debug (sessions) 0 0 0
Test Status (with successful) 0 0 0
Test Status (without successful) 0 0 0
Coverage Engines
JetBrains dotCover 2 (0%)
Agent Java Versions
1.7 1 (50%)
1.8 1 (50%)
Web Pages Usage
Week Month
/agentDetails.html 2 (100%) 2 (40%)
/agentDetails.html?tab=agentCompatibleConfigurations 2 (100%) 2 (40%)
/agentDetails.html?tab=agentSummary 1 (50%) 1 (20%)
/allProjects.html 1 (50%) 2 (40%)
/changesPopup.html 1 (50%) 3 (60%)
/diffView.html 0 (0%) 1 (20%)
/filesPopup.html 1 (50%) 2 (40%)
/overview.html 2 (100%) 4 (80%)
/project.html 1 (50%) 2 (40%)
/project.html?tab=projectChangeLog 1 (50%) 1 (20%)
/project.html?tab=projectOverview 1 (50%) 3 (60%)
/promoDetailsPopup.html 1 (50%) 1 (20%)
/remoteAccess/rdp.html 1 (50%) 1 (20%)
/runCustomBuild.html 1 (50%) 3 (60%)
/systemProblems.html 1 (50%) 3 (60%)
/viewArtifactsPopup.html 1 (50%) 4 (80%)
/viewLog.html 1 (50%) 2 (40%)
/viewLog.html?tab=buildLog 1 (50%) 3 (60%)
/viewLog.html?tab=buildResultsDiv 1 (50%) 2 (40%)
/viewLog.html?tab=Duplicator 0 (0%) 1 (20%)
/viewLog.html?tab=Inspection 0 (0%) 1 (20%)
/viewModification.html?tab=vcsModificationFiles 1 (50%) 1 (20%)
/viewType.html 1 (50%) 3 (60%)
/viewType.html?tab=buildTypeChangeLog 1 (50%) 1 (20%)
NuGet Versions
2.8.5 43 (100%)
5 comments
Comment actions Permalink

Hi Niels,

Please follow the recommendations from the guideline: https://confluence.jetbrains.com/display/TCD9/Reporting+Issues#ReportingIssues-OutOfMemoryProblems.
If memory settings tuning does not help, then please attach memory dump.

0
Comment actions Permalink

We are running with -Xmx1200m -XX:MaxPermSize=270m

After running a cleanup, the issue seems to have gone away - memory consumption is now down:
Total heap: 732.95 MB (69% of maximum available 1.04 GB)
Data: 547.52 MB (68% of maximum available 800 MB)

I assume that the cleanup has flushed some of the old build info from the caches.
I will not do more about for now but watch for further problems.

Thanks!

0
Comment actions Permalink

Glad it works! It is recommended to configure clean-up rules to remove obsolete builds and their artifacts, purge unnecessary data from database and caches in order to free disk space, remove builds from the TeamCity UI and reduce the TeamCity workload. And to run clean-up on regular basis.

0
Comment actions Permalink

Unfortunately, the server is acting up again.
I have uploaded a memory dump and some thread dumps to  ftp://ftp.intellij.net/.uploads
- the filename is teamcity_memory_dumps_bruel_og_kjaer.zip

0
Comment actions Permalink

It looks like your server produces fair amount of builds every day. I'd recommend configuring stricter cleanup rules in this case or switch to 64bit JVM and give TeamCity server more memory.
Also in upcoming 9.1.1 bugfix release we added some related memory usage improvements It should be published on the next week.

0

Please sign in to leave a comment.