Migration to new server and new database – indexing stuck during restore

Hello TeamCity support,

We are migrating our TeamCity server to a new Windows Server 2022 host and switching the database from MSSQL to PostgreSQL 15 (running inside a Docker container).

The server is restored from a backup created through the Web UI (basic backup, excluding build artifacts).

After restoring the backup, the system starts reindexing the metadata and local search indexes. However, the indexing seems to get stuck around 80–90% (e.g., "3110/3593 tasks completed"), and no visible progress is made for hours.

We tried waiting for a very long period — more than two full days (the entire weekend) — but indexing did not progress at all. No errors were observed in the logs (teamcity-server.log and teamcity-rest.log).

We have tried restoring both to PostgreSQL and MSSQL, and in both cases the issue appears similarly: indexing during the first server start after restore freezes and no further activity happens (no errors in logs, just stuck on "sync in process").

Environment:
- TeamCity 2024.03.3 (build 156364)
- PostgreSQL 15 in Docker (official image)
- 8 GB RAM assigned to the server
- JVM heap size currently set to 4 GB (-Xmx4096m)

Based on PostgreSQL monitoring (pgAdmin dashboards), we can see that database activity has dropped to near zero, confirming that TeamCity is no longer actively progressing with the indexing process.

Questions:
- Is it normal for indexing to take multiple hours or even days after a database restore?
- Even with 4 GB heap size, is there anything else we should adjust to improve the indexing performance?
- Is there a way to safely disable or defer the indexing process to bring the server online faster?

Thank you in advance!
 

0
1 comment

Depending on the size of the TeamCity installation and several other performance factors, such as disk IO speed, indexing can take a while. However, several days would certainly seem longer than what we would typically expect. In this case, I would suggest checking for any server thread dumps to see if there is some task that is stuck or blocked. Please find the information on server thread dumps in our documentation here: https://www.jetbrains.com/help/teamcity/reporting-issues.html#Server+Thread+Dump.

It may also be worth restarting your TeamCity service to see if that helps move things along. 

0

Please sign in to leave a comment.