Managing memory usage
Hi there,
I have a 'Low Pool Memory' problem with my TeamCity instance, and I want to get some advice on what I had configured wrongly.
Hardware specification for TeamCity Server:
Model: Intel Xeon 2.40 GHz
Core: 2
Memory: 4GB
TeamCity Usage Statistics:
Connected Agents: 2
Build Configurations: 90
Build Count: 525 (daily)
I am running the 64-bit version of TeamCity and memory settings are as follows:
TEAMCITY_SERVER_MEM_OPTS=-Xmx1638m -XX:ReservedCodeCacheSize=350m
I am wondering if I need to run my server on a bigger memory hardware?
But, based on what I read on https://confluence.jetbrains.com/pages/viewpage.action?pageId=113084582#HowTo...-chooseOSForServer, generally, I don't need to dedicate more than 4gb of memory if I don't plan to use more than 100 concurrent build agents and more than 200 users.
Thanks for the help :)
Please sign in to leave a comment.
Hello,
Judging by these charts, it does not look like you're using 64-bit JVM or Xmx is set to 1638m. You can double check it on the same diagnostics page. See Java Configuration section there.
Hi Pavel,
Thanks for your reply. I could see that Xmx being set to 512m under the Java Configuration. Is this the true representation of what I have set as the Xmx value?
I have made several attempts at changing this value by passing TEAMCITY_SERVER_MEM_OPTS as the environment variable:
a. Passing TEAMCITY_SERVER_MEM_OPTS to /etc/environment
Steps:
1. Add TEAMCITY_SERVER_MEM_OPTS=-Xmx1638m -XX:ReservedCodeCacheSize=350m on /etc/environment file.
2. Restart TeamCity Server
I confirmed that this environment variable is being set properly by running 'env' command to check for all environment variables and I can see that TEAMCITY_SERVER_MEM_OPTS is now being passed as what I had provided above.
I had also echo this variable on teamcity-server.sh file and it printed the value that I set on /etc/environment.
b. Passing TEAMCITY_SERVER_MEM_OPTS through TeamCity Internal Properties
Steps:
1. Navigate to Administration > Diagnostics > Internal Properties
2. Edit Internal Properties
3. Pass TEAMCITY_SERVER_MEM_OPTS=-Xmx1638m -XX:ReservedCodeCacheSize=350m
However, even after these attempts Xmx value under Java Configuration does not change. Can you please provide a guide on updating this value?
Thanks,
Anto
Hi Anto,
The memory settings have to be set up before teamcity starts, as they are used to run the JVM. This means that adding them as internal properties will not work, as they are read when the JVM is already started. B will thus not work.
For your "A" solution, it depends on how you are running the server's process. /etc/environment is read by shells, but it might not be read by some process handlers unless explicitly stated, which means that you might run teamcity-server.bat manually, see the value, but then restart it with, say, systemd, and it wouldn't be loaded. If you are running it from the command line, and the spawned process is still showing -Xmx512, please send us the exact commands you use, and the exact setup you have so that we can try and replicate the issue.
Hi Denis
Thanks a lot for your help on this and pointed out that if a service is run through systemd, /etc/environment won't be loaded.
I passed 'TEAMCITY_SERVER_MEM_OPTS' variable to systemd and confirmed that Xmx value is now been updated.
I'm having a similar issue but I'm running on a Windows server, teamcity is still using it's 32bit jre, and it's run as a windows service under a domain admin account.
I've set System Variable and user variable for the admin user of TEAMCITY_SERVER_MEM_OPTS=-Xmx1024m
The server was rebooted and verified it was set
But everytime the TeamCity starts up it passes an java arg of -Xmx512m
[2019-01-14 13:54:43,045] INFO - jetbrains.buildServer.STARTUP - Java: 1.8.0_161, Java HotSpot(TM) Server VM (32 bit) (25.161-b12, mixed mode), Java(TM) SE Runtime Environment (1.8.0_161-b12), Oracle Corporation; JVM parameters: -Djava.util.logging.config.file=D:\TeamCity\bin\..\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Xrs -Xmx512m -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j.configuration=file:../conf/teamcity-server-log4j.xml -Dteamcity_logs=D:\TeamCity\bin\..\logs -Dignore.endorsed.dirs= -Dcatalina.base=D:\TeamCity\bin\.. -Dcatalina.home=D:\TeamCity\bin\.. -Djava.io.tmpdir=D:\TeamCity\bin\..\temp
Hi Brian,
If you stop the server, then run it manually from the command line (access the folder, run teamcity-server.bat start), does it pick up the proper -Xmx? If yes, then it's an issue of where you have set up the variables for the admin user. The windows service might not be picking them up properly.
Thanks for the response Denis, unfortunately, I never had the chance to try it. The good news is sometime in the last 9 months the TC service started to pick up the -Xmx change. We recently updated to 2019.1.3 so maybe that was it.