TeamCity hangs every few days

Hi,

TeamCity version - TeamCity Professional 2024.12.2 (build 174504)

Linux - Ubuntu 24.04.2 LTS

Proxy - Nginx

Symptoms:

Every few days TeamCity server hangs (I can see 502 nginx error) and generating thread dump with header:

Server cannot shutdown properly: some non-daemon threads are still running

Upload id: 2025_02_13_p7dLbSXLbXsNP5Xv9UwVRH (file: 2025-02-13_07.11.10-shutdown.txt)

Any suggestions ?

0
3 comments

Hi,

For performance issues, such as slowness or delays, please collect a set of at least 10 thread dumps (spread across the issue timeframe) from the server or agent during the occurrence of the issue. You can refer to this guide for instructions. Ensure that you send us the dumps along with relevant data.

From the thread dump, we can observe the following details you mentioned:
Threads related to Tomcat (http-nio-8111-exec-*), Amazon EC2, and JGit are still active. These are likely waiting on network tasks, request processing, or cloud agent synchronization.
Check the TeamCity Web UI for any running builds and manually stop them if necessary.
 

> The server cannot shut down properly due to non-daemon threads still running:

CPU Load Analysis

From the log file, I noted high CPU load due to the following details:

  1. Load average (1m) = 2.52 → On average, 2.52 processes are waiting for CPU execution in the last minute.
  2. Overall CPU usage = 70.65%
  3. TeamCity process CPU usage = 2.10% → TeamCity itself is consuming minimal CPU, indicating that other processes are responsible for most of the load. Consider identifying and analyzing high-CPU-consuming processes.

Memory Usage Analysis

G1 Old Gen:
Used = 463.18 MB
Committed = 536 MB
Max = 1 GB
Used/Max = 45.23%
Total Memory:
Used = 1.15 GB
Max = 2.62 GB
Used/Max = 44.13%

While memory usage is not an immediate issue, prolonged garbage collection pressure could develop. The G1 Old Gen (45% usage) suggests that memory may not be efficiently reclaimed.

To optimize memory allocation, please follow the TeamCity Server Requirements and adjust the -Xmx memory value accordingly.

Best Regards,

Tom

0

I forgot to tell that TeamCity server hangs after independently restarts.

0
Hi

Please collect a set of at least 10 thread dumps (spread across the issue timeframe) from the server or agent during the occurrence of the issue. You can refer to this guide (https://www.jetbrains.com/help/teamcity/reporting-issues.html#Server+Thread+Dump) for instructions.

Please also share the teamcity-server.log with us for further investigation.

Best Regards,
Tom
0

Please sign in to leave a comment.