TeamCity Server on Windows crashes and generates minidump files after update to 2023.11.4
We just updated from 2023.05 to 2023.11.4 and after the update was complete the TeamCity service crashed three times, then stayed up and running. The next morning, we did some disk cleanup which involved restarting the TeamCity service and it has been crashing since and not recovering. We cleaned up the core dump files from the previous day, the .old directory from the update, and all files from the F:\TeamCity\temp directory (after stopping TeamCity).
Here is some information from the dump log file.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffb9e5a1c16, pid=4608, tid=1856
#
# JRE version: OpenJDK Runtime Environment Corretto-11.0.9.12.1 (11.0.9.1+12) (build 11.0.9.1+12-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-11.0.9.12.1 (11.0.9.1+12-LTS, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
# Problematic frame:
# V [jvm.dll+0x1a1c16]
#
# Core dump will be written. Default location: F:\TeamCity\bin\hs_err_pid4608.mdmp
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
--------------- S U M M A R Y ------------
Command Line: --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=F:\TeamCity\bin\..\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Xmx7G -XX:ReservedCodeCacheSize=450m -Xrs -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j2.configurationFile=file:../conf/teamcity-server-log4j.xml -Dteamcity_logs=F:\TeamCity\bin\..\logs -Dignore.endorsed.dirs= -Dcatalina.base=F:\TeamCity\bin\.. -Dcatalina.home=F:\TeamCity\bin\.. -Djava.io.tmpdir=F:\TeamCity\bin\..\temp -agentpath:C:\Program Files\SentinelOne\Sentinel Agent 23.2.3.358\SentinelJava64.dll=extendedCapabilities org.apache.catalina.startup.Bootstrap service start
Host: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz, 4 cores, 7G, Windows Server 2019 , 64 bit Build 17763 (10.0.17763.4974)
Time: Fri Mar 15 07:11:25 2024 Eastern Daylight Time elapsed time: 305.295993 seconds (0d 0h 5m 5s)
...
--------------- S Y S T E M ---------------
OS: Windows Server 2019 , 64 bit Build 17763 (10.0.17763.4974)
OS uptime: 0 days 0:05 hours
VMWare virtualization detected
CPU:total 4 (initial active 4) (2 cores per cpu, 1 threads per core) family 6 model 85 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, tsc, tscinvbit, bmi1, bmi2, adx, evex, fma
Memory: 4k page, system-wide physical 8191M (3338M free)
TotalPageFile size 9471M (AvailPageFile size 4973M)
current process WorkingSet (physical memory assigned to process): 2299M, peak: 2309M
current process commit charge ("private bytes"): 2289M, peak: 2299M
vm_info: OpenJDK 64-Bit Server VM (11.0.9.1+12-LTS) for windows-amd64 JRE (11.0.9.1+12-LTS), built on Nov 4 2020 10:17:53 by "Administrator" with MS VC++ 15.9 (VS2017)
We have rebooted the host (VM), but not sure what's going on.
Please sign in to leave a comment.
We just upgraded the version of Amazon Corretto to 17.0.10.7.1 and are still experiencing the crashes. It seems like if we get TeamCity to load, but navigating to the Administration page causes the crash.
Update: CPU utilization spikes to 100% when the crashes happen.
Discovered that the problem was giving the JVM more than half the memory the hosting system (VM) had. We were giving the TeamCity JVM 7GB when the VM only had 8GB available. This can be closed.