Unexpected Restart after an EOF exception
Dear forum,
We had an unexpected server restart. Here is the server log from that period.
I only redacted an IP from the log steam. I didn't remove any lines for the 8 minute span.
A few questions
1. Is the restart related to the ERROR just before? It's 40 seconds apart… a bit long for a crash; but it is also the literal last event preceding.
2. What does the error mean? java.io.EOFException; while processing request: POST ‘/ajax.html’. Did the user disconnect? (and possibly crash the server?)
[2025-04-12 02:16:30,329] INFO - jetbrains.buildServer.AGENT - Removing agent Unregistered agent "tc-64-agent2" {id=478}
[2025-04-12 02:16:30,354] INFO - jetbrains.buildServer.AGENT - Agent "tc-64-agent2" {id=478} has been unregistered. Reason: Agent has unregistered
[2025-04-12 02:21:31,302] INFO - jetbrains.buildServer.AGENT - Removing agent Unregistered agent "tc-64-agent1" {id=477}
[2025-04-12 02:21:31,322] INFO - jetbrains.buildServer.AGENT - Agent "tc-64-agent1" {id=477} has been unregistered. Reason: Agent has unregistered
[2025-04-12 02:22:52,426] ERROR - jetbrains.buildServer.SERVER - Error java.io.EOFException; while processing request: POST '/ajax.html', from client {REDACTED IP4} ({REDACTED IP4}:{REDACTED PORT}), client: AJAX-Prototype, authenticated as user with id=41java.io.EOFException
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1322)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1233)
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:796)
at org.apache.coyote.http11.Http11InputBuffer.access$400(Http11InputBuffer.java:41)
at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1202)
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:96)
at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:244)
at org.apache.coyote.Request.doRead(Request.java:626)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:303)
at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:615)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:358)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:132)
at org.apache.catalina.connector.Request.readPostBodyFully(Request.java:3072)
at org.apache.catalina.connector.Request.parseParameters(Request.java:2987)
at org.apache.catalina.connector.Request.getParameter(Request.java:1088)
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:309)
at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:4)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1793)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Thread.java:1583)
[2025-04-12 02:23:33,208] INFO - jetbrains.buildServer.AGENT - Removing agent Unregistered agent "tc-32-agent1" {id=479}
[2025-04-12 02:23:33,231] INFO - jetbrains.buildServer.AGENT - Agent "tc-32-agent1" {id=479} has been unregistered. Reason: Agent has unregistered
[2025-04-12 02:24:35,571] INFO - jetbrains.buildServer.STARTUP - Path to the startup configuration file: /opt/teamcity/bin/../conf/teamcity-startup.properties
[2025-04-12 02:24:36,031] INFO - jetbrains.buildServer.SERVER - Using logs directory "/opt/teamcity/logs"
[2025-04-12 02:24:36,188] INFO - jetbrains.buildServer.STARTUP -
[2025-04-12 02:24:36,214] INFO - jetbrains.buildServer.STARTUP - ===========================================================
[2025-04-12 02:24:36,231] INFO - jetbrains.buildServer.STARTUP - Starting TeamCity server
[2025-04-12 02:24:36,750] INFO - jetbrains.buildServer.STARTUP - TeamCity version: 2025.03 (build 186049), data format version 1021
[2025-04-12 02:24:36,778] INFO - jetbrains.buildServer.STARTUP - OS: Linux, version 5.15.173.1-2.cm2, amd64, Current user: tcuser, Time zone: EDT (UTC-04:00)
Please sign in to leave a comment.
Short answer: probably not directly. The java.io.EOFException happened at 02:22:52.
The TeamCity startup log begins at 02:24:35. That’s a 1 minute 43 second gap between the error and startup — fairly long for a hard crash, but not unheard of if the system was slow or waiting on timeouts/shutdown hooks.
EOFExceptions alone usually don't crash the server. This might be related to potential resource issues (such as insufficient memory, full disk space, etc.)
Could you please share the teamcity-server.log file (https://www.jetbrains.com/help/teamcity/teamcity-server-logs.html) that covers the timestamp when the issue occurred? Files can be uploaded via https://uploads.jetbrains.com/. Please let us know the exact id after the upload.
Best Regards,
Tom
Hi Tom. Thanks for offering to look into it. Please let me know what you see around that time period.
Hi Joshua,
It is highly likely that the TeamCity restart on April 12 was triggered by resource constraints, most likely due to the insufficient memory (2GB RAM). After the restart, the logs show no further restarts, indicating that the server remained stable under the current load.
Running TeamCity on a server with only 2GB of RAM is critically low, especially under the following conditions:
- Multiple build agents are connected
- Builds are running or queued
- Several plugins are loaded
- The web UI is being accessed concurrently
Typical symptoms of low memory include:
- Random agent disconnections or registration
- Unexpected EOF Exception errors or Tomcat input buffer failures
- Slow or unresponsive UI, request parsing failures (as seen with /ajax.html)
- Silent crashes or hangs that may trigger automatic restarts (if a watchdog is configured)
To ensure a stable and reliable operation, it is strongly recommended to upgrade the server memory to at least 4–8 GB or more if lots of agents or plugins are in use.
For guidance on configuring memory settings in TeamCity, please refer to the official documentation:
Configure Memory Settings for TeamCity Server.
Best Regards,
Tom
Tom, thank you so much for looking into this. I confess I run this based on the default behaviour of Teamcity's docker image. and I never noticed it is only at `-Xmx2g` until you pointed it out just now.
Based on the content from teamcity-docker-images/dockerhub/teamcity-server/README.md at master · JetBrains/teamcity-docker-images, I plan to add `-e TEAMCITY_SERVER_MEM_OPTS="-Xmx6g"` environment variable to raise the allocation to 6GB.
Are there any other parameters from the default docker set that's worth adjusting?
You're very welcome.
Yes, increasing the heap size to 6GB with TEAMCITY_SERVER_MEM_OPTS="-Xmx6g" is a good move, especially if your server handles a fair amount of a load or large build histories.
In terms of other parameters, it really depends on your setup. For more suggestions and details about CPU, disk and network-related information, please refer to https://www.jetbrains.com/help/teamcity/system-requirements.html#TeamCity+Server+Requirements.
Best Regards,
Tom