java.lang.OutOfMemoryError: unable to create new native thread

Hi,

During the last couple of months our TeamCity server have seen an increase in memory related crashes. Lately it's been happening almost daily and is becoming a bit of an annoyance. These are the two exceptions that always show up:

[2020-08-19 14:12:26,959] ERROR - jetbrains.buildServer.SERVER - Got unexpected exception, thread "http-nio-80-ClientPoller-1" is terminated. Error: java.lang.OutOfMemoryError: unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:167)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:145)
at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1080)
at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:889)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:865)
at java.lang.Thread.run(Thread.java:748)
and

[2020-08-19 14:21:39,205] WARN - r.configs.dsl.DslPluginManager - DSL documentation generation failed
exit code: 1
stderr: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at org.jetbrains.dokka.HtmlFormatService.appendOutlineHeader(HtmlFormatService.kt:124)
at org.jetbrains.dokka.OutlineFormatService$DefaultImpls.appendOutline(OutlineService.kt:17)
at org.jetbrains.dokka.HtmlFormatService.appendOutline(HtmlFormatService.kt:111)
at org.jetbrains.dokka.OutlineFormatService$appendOutline$1.invoke(OutlineService.kt:21)
at org.jetbrains.dokka.OutlineFormatService$appendOutline$1.invoke(OutlineService.kt:8)
at org.jetbrains.dokka.HtmlFormatService.appendOutlineLevel(HtmlFormatService.kt:129)
at org.jetbrains.dokka.OutlineFormatService$DefaultImpls.appendOutline(OutlineService.kt:20)
at org.jetbrains.dokka.HtmlFormatService.appendOutline(HtmlFormatService.kt:111)
at org.jetbrains.dokka.OutlineFormatService$appendOutline$1.invoke(OutlineService.kt:21)
at org.jetbrains.dokka.OutlineFormatService$appendOutline$1.invoke(OutlineService.kt:8)
at org.jetbrains.dokka.HtmlFormatService.appendOutlineLevel(HtmlFormatService.kt:129)
at org.jetbrains.dokka.OutlineFormatService$DefaultImpls.appendOutline(OutlineService.kt:20)
at org.jetbrains.dokka.HtmlFormatService.appendOutline(HtmlFormatService.kt:111)
at org.jetbrains.dokka.OutlineFormatService$appendOutline$1.invoke(OutlineService.kt:21)
at org.jetbrains.dokka.OutlineFormatService$appendOutline$1.invoke(OutlineService.kt:8)
at org.jetbrains.dokka.HtmlFormatService.appendOutlineLevel(HtmlFormatService.kt:129)
at org.jetbrains.dokka.OutlineFormatService$DefaultImpls.appendOutline(OutlineService.kt:20)
at org.jetbrains.dokka.HtmlFormatService.appendOutline(HtmlFormatService.kt:111)
at org.jetbrains.dokka.OutlineFormatService$DefaultImpls.formatOutline(OutlineService.kt:28)
at org.jetbrains.dokka.HtmlFormatService.formatOutline(HtmlFormatService.kt:96)
at org.jetbrains.dokka.FileGenerator.buildOutlines(FileGenerator.kt:55)
at org.jetbrains.dokka.GeneratorKt.buildAll(Generator.kt:14)
at org.jetbrains.dokka.GeneratorKt.buildAll(Generator.kt:23)
at org.jetbrains.dokka.DokkaGenerator.generate(DokkaGenerator.kt:47)
at org.jetbrains.dokka.MainKt.entry(main.kt:132)
at org.jetbrains.dokka.MainKt.main(main.kt:177)

We are running a small setup with just 5 build agents and ~30 developers running around 100 builds per day hence we are still running the 32-bit version of the Teamcity server. The Java process is running with the max memory as instructed by the documentation (1200MB) but we are still seeing crashes. The crashes are random and sudden which means we've been unable to create a memory dump as the memory usage increases. The regular memory usage during normal operation is between 30% and 60%.

TeamCity is running on an AWS EC2 host with 8GB of RAM and 2 CPUs.

We have been uprading the Teamcity software on a regular basis and we are currently running the 2020.1.3 (build 78866) version.

One thing of importance to mention is that we during the month of July moved parts of our source code from BitBucket to GitHub which means we now have more VCS roots configured in the TeamCity Server. We expect that to use more resources but have a hard time seeing how it would consume all available memory.

Attached to the case you will find logs of a crash with debug-general active.

Upload id: 2020_08_20_B6ARZzsiJi9Eh1W4 (file: logs.zip)