Teamcity docker upgrade fails
I use docker containers for teamcity server and trying to upgrade from v 2017.1 to 2017.2. Documentation states that I don't need to perform any additional steps, just pull new container and run it. But when I did this, I've got and error message:
Data parts are inconsistent: the Data Directory exists (from another version of TeamCity) but the database does not.
(I use default HQL database)
I tried to restore HQL database inside the docker container using `maintainDB.sh` command, but when I tried to do it, I've got a message that HQL database already used (possible because teamcity server is running)
I tried to shutdown teamcity server using `teamcity-server.sh stop` command, but when I ran it docker container exited and I have no way to run `maintainDB.sh` script on stopped container.
Is it possible to upgrade teamcity from v2017.1 to v2017.2 when database in HQL using docker containers? Is it possible to restore database backup when teamcity runs in docker? If it's possible, how to do this.
Please sign in to leave a comment.
Hi Sergey,
exactly which image are you using? Windows/Linux? The HSQL integrated database resides in the data directory, so if the data directory was conserved, the database should still be there, and upon starting the server in a newer version it should prompt you to upgrade. Could you confirm that the database files (buildserver.* within the <data_directory>/system) are still present and haven't been recreated by starting the new one?
Hi, Denis,
I'm using linux host for docker. Do you mean `buildserver.properties` and `buildserver.script` files? In version 2017.1 I see only those two files. How can I check that the files are recreated from scratch but not just updated during upgrade process?
Hi Sergey, sorry for the delay, had a few rough days. Those 2 files are part of the database, but not all that are required. You can check creation vs update date, the files shouldn't be recreated on each start, but I wanted to know if the upgrade removed them, and it seems there is at least a couple missing (buildserver.data, .log, and usually a .tmp folder). The files missing are obviously what's telling the server they don't exist, and teamcity should not have deleted them, this is definitely not normal behavior. The ".data" file missing in particular is worrying, as it should be the file holding all the actual data from the database, so if the file got deleted, it's likely you lost all the database information. It would be important to understand what, how and why happened with this file. If your file system can keep track of it, I would definitely suggest trying.
A couple extra pointers:
-Using the HSQL database is not recommended for production purposes, just for test. It has a number of limitations and, while the process should work normally, it might have issues at times: https://confluence.jetbrains.com/display/TCD10/Setting+up+an+External+Database
-The error message seems to point out that the buildserver files aren't present (or at least some of them). The update process should update both, make sure the files have appropriate permissions
-The maintaindb script is used to perform backups, and this is recommended obviously with the server stopped, but docker presents a problem when performing this actions. A pretty simple solution is to simply have a local installation that you can use for maintenance when the main server has to be stopped.
I don't get the last sentence about restoring from backup. If I want to backup and restore database in dockerized environment then I have to remove docker and reinstall teamcity server without docker? Did I correctly understand that teamcity server in docker container does not allow to restore database from backup?
I think this is a valid question. I am also just experiencing the same and I am kind of afraid this may indeed be the case. Does it really mean that when we use TeamCity through Docker you cannot use your backups across different versions? I hope this is not the case and I am mistaken. Please confirm :)
Hi Prasath,
could you specify what your current problem is? I think I misspoke on that sentence a bit but it was expected to be taken into the full context of the ticket.
TeamCity can restore backups when using docker, the problem here is a large combination of things:
-Upgrade using the internal database (which is only meant for test purposes, if you plan on using TeamCity in production don't use the internal database)
-Some as-of-yet unknown issue that removed several of the database files, probably during the upgrade process, but still not certain.
-The requirement to restore an old backup using this setup under this conditions.
Hi Denis, I provided much more details in this ticket: https://teamcity-support.jetbrains.com/hc/en-us/community/posts/360000300500
* I am not using internal database, using postgresql which runs in a separate docker container
* I am ok when I have to start with a empty database. I need to take over the build configurations
* If you asking for my goal: It is as simple as upgrade 2017.1.3 to 2018.1.1 and keep rolling. Until now upgrading using Docker container is NOT possible using various techniques all outlined in the above mentioned post. I'm praying that there is a way to upgrade TeamCity on Docker, but was not able yet to see in release notes and documentation how that migration needs to be executed.
Any help on this issue is much appreciated. Thank you.