Version mismatch NIGHTMARE: Data format of the data directory (1003) and the database (1008) do not match.

Avatar
Mostefa Kamal-lala
Created

Hello TeamCity Support,

We are running TeamCity Server in a container. Yesterday, the TeamCity server pod restarted due to maintenance in our Kubernetes cluster. The server had been running without issues since April 19, 2024. However, after the restart, the server failed to start due to a version mismatch between the database and the data directory.

The server's Docker image is version 2023.11.4. In our MSSQL external database, the db_version table shows the following:

version_number    version_time    incompatible_change
994    2023-08-14 09:05:07.713    1
995    2024-04-18 10:02:04.027    1
996    2024-04-18 10:02:04.160    1
997    2024-04-18 10:02:04.270    1
998    2024-04-18 10:02:04.450    0
999    2024-04-18 10:02:04.483    0
1000    2024-04-18 10:02:04.547    0
1001    2024-04-18 10:02:04.677    0
1002    2024-04-18 10:02:04.760    0
1003    2024-04-18 10:02:04.787    0
1004    2024-04-18 11:52:13.673    0
1005    2024-04-18 11:52:17.430    0
1006    2024-04-18 11:52:17.507    0
1007    2024-04-18 11:52:17.593    0
1008    2024-04-18 11:52:17.673    0

On the maintenance page, it shows:

  • Software version: 1003
  • Data directory version: 1003
  • Database version: 1008

How can we resolve this issue without losing data? The oldest backup we have is almost two months old.

Could the issue stem from the server initially starting with version 2024.03 and then being restarted with version 2023.11.4 while using the same persistent data and database? If so, why did the server run without issues until the recent restart?

Thank you for your assistance.

Votes

1

Share

4 comments
Avatar
Mikhail Efremov
Created
This issue is being handled via a private channel.
1
Avatar
Anatoly Cherenkov
Created

wufer Manually editing the database is highly unrecommended, and I'm afraid it won't help. The database format will stay the same, but the format version number will change, making the database inconsistent.

The 'Version mismatch' error occurs because TeamCity's <data directory>/config/database.properties file points to the database that has the format used by TeamCity 2024.03.x. (format version 1008), while the data directory specified in the <TeamCity home directory>/conf/teamcity-startup.properties file has the format used in TeamCity 2023.05 (format version 1003). For TeamCity to be able to start, both the data directory and the database need to have the format of the same version.

You can recover from the 'Version mismatch' error if you still have a valid backup from the old version:

  1. Install TeamCity from scratch (same version as the version inside the backup file or newer);
  2. When, on the first start, it asks for the data directory location, you can use the Restore from backup option. When restoring from the backup, you need to specify an empty data directory and a preemptively created *empty* database.

You can see more context in the documentation here.

0
Avatar
Anatoly Cherenkov
Created
First of all, please note that if you see the Version Mismatch error, it typically means you are trying to use the database and data directory that don't belong together (for example, they come from different installations or are taken from the same installation but at different points in time). The most recommended actions are either to restore from a backup or use a consistent database and data directory pair.

If the upgrade finished successfully, there should exist a data directory that has the same format version as the database. Consider looking it up. The `<data_dir>/config/main-config.xml` file would have `<version number="1008" />` in it (be sure to never change the version manually).

Also, when you commit to upgrading the TeamCity database and data directory, TeamCity automatically creates a backup of those (unless you opt out of it). You can try searching for the backup created at the upgrade time. The default backup filename is `<data_dir>/backup/TeamCity_Before_Upgrade_<date>_<time>.zip`.

If there is no backup, and the data directory upgraded to the new format is not there for whatever reason, install your previous TeamCity version on a separate machine, restore the data directory on it, and perform the upgrade there on top of an empty database. That will bring your data directory to the newer version's format, which you can then simply move to the original server. You might need to drop the `<data_dir>/system/caches` directory before starting the server.
0
Avatar
Mostefa Kamal-lala
Created

FYI, I did what was not recommended: Modifying the versions table in the database.
However, I made sure I had a copy of the database and data directory to play with to avoid any data loss.
The reason why I did so is because I have followed properly all the steps in the previous upgrade and everything was running fine so I assumed for an obsur reason some steps were not completed while the data directory structure was successfuly upgraded resulting in the successful upgrade message.

I know it is not recommended but I had validated everything and it is running just fine. Keep in mind, if you want to try this, do it with a copy of the database and data directory to test it out.

0

Please sign in to leave a comment.