DSL process failed: There is insufficient memory for the Java Runtime Environment to continue.

Permanently deleted user

Created March 02, 2021 14:02

Hi,

we are running TeamCity 2020.2.2 inside a Windows docker container. The container is configured to use a maximum of 4GB of memory (host has 32GB of memory, ~23GB are not in use).

There are 5 projects (with 12 build types). All projects are configured to use versioned settings in Kotlin DSL

From time to time the DSL process fails and an OutOfMemory error is shown in the build log:

```

07:43:58 Finalize build settings

07:43:58 Retrieve settings, revision: 173ef3de45fa

07:43:58 Settings VCS root details

07:43:58 Load settings from VCS

07:44:45 Generate settings, format: kotlin

07:44:46 DSL process failed

exit code: 1

stdout: #

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (mmap) failed to map 536870912 bytes for Failed to commit area from 0x00000000e0000000 to 0x0000000100000000 of length 536870912.

# An error report file with more information is saved as:

# C:\TeamCity\temp\configs12682866081504828698dsl\hs_err_pid11808.log

stderr: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e0000000, 536870912, 0) failed; error='The paging file is too small for this operation to complete' (DOS error/errno=1455)

```

The last line, saying that the paging file is too small, occurs only occasionally. Most of the time it looks like this:

```

13:51:11 Finalize build settings

13:51:11 Retrieve settings, revision: d99092ba054b

13:51:11 Settings VCS root details

13:51:11 Load settings from VCS

13:52:04 Generate settings, format: kotlin

13:52:14 DSL process failed

exit code: 1

stdout: Attempt to store a default value of TeamCity internal property before initialization!

Attempt to read TeamCity internal property "teamcity.xmlParser.cacheGrammars" before initialization! Default value is used. Callee: jetbrains.buildServer.util.XMLParser.createSAXBuilder(XMLParser.java:69)

Attempt to store a default value of TeamCity internal property before initialization!

Attempt to read TeamCity internal property "teamcity.kotlinConfigsDsl.loadBundledDsl" before initialization! Default value is used. Callee: jetbrains.buildServer.configs.dsl.DslPluginManager.parseServerJarExtensions(DslPluginManager.java:865)

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 764976 bytes for Chunk::new

# An error report file with more information is saved as:

# C:\TeamCity\temp\configs15462762110080025577dsl\hs_err_pid11632.log

[thread 6668 also had an error]

# Compiler replay data is saved as:

# C:\TeamCity\temp\configs15462762110080025577dsl\replay_pid11632.log

stderr: WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by com.intellij.util.ReflectionUtil (file:/C:/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/configs-dsl/kotlin_runtime/lib/kotlin-compiler.jar) to method java.util.ResourceBundle.setParent(java.util.ResourceBundle)

WARNING: Please consider reporting this to the maintainers of com.intellij.util.ReflectionUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

```

Unfortunately the log files (`hs_err_pid*.log` and `replay_pid*.log`) are not present (C:\TeamCity\temp is bound to a volume of the host system, TC is deleting the files right after they appear).

The diagnostics page says, that the server is using about 350MB of memory, `TEAMCITY_SERVER_MEM_OPTS` is set to `-Xmx3g -XX:ReservedCodeCacheSize=350m`.

I also logged the memory consumption of all java processes running inside the container. When triggering 3 build configurations at the same time, it looks like this (at the time, the memory allocation error occurs):

The first two DSL processes (PID 14328 and 8816) are running fine, the third one (PID 11808) crashes. It is the first and only time, the process occurs in the log. So it seems like it is crashing right after startup.

`docker stats` says, that the container is not using more than ~2.5 GiB of memory in total.

Any idea why there is not enough memory for the DSL process or how we could further investigate this (why is TeamCity deleting the error log files)?

Thanks for your help

1 comment

Trifan Stanislav

Created May 22, 2021 19:32

Having same issue. Having no pagefile at all. 32 GB of total memory where about 10GB available.
Having a simple maven java project with few modules (very light ones). Oracle OpenJava 16 on win 10 x64.
Getting next exception when try to run a test from IDE:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000602400000, 536870912, 0) failed; error='The paging file is too small for this operation to complete' (DOS error/errno=1455)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 536870912 bytes for G1 virtual space

Please sign in to leave a comment.