Backing Up 6.5 to migrate to 7.x

Hello,

I'm new to administering TeamCity.
Our CentOS 5.5 server running TeamCity 6.5 is running critically low on disk space.

I am currently brining up a new CentOS server with TeamCity 7.  I need to understand a little bit more about what needs to come over from the old server to the new server.

I realize that I can use the webgui to perform a backup that will place files in /home/teamcity/.BuildServer/backup

My concern is that I'm not sure it gets everything.  We have about 98 gigs of files in the /home/teamcity/.BuildServer/system/artifacts folder.  There are also about 36 megs worth of build-specific files in /home/teamcity/.BuildServer/config

Do I need to just manually scp those files from the old server to the same folders inside .BuildServers on the new server?  Along with performing a backup with scope "All Except Build Artifacts"?

Then use the dbmaintenance script to do a restore?

I'm actually worried that the backup through the webgui is going to fail due to low disk space.

Any help is greatly appreciated.

Thanks

13 comments
Comment actions Permalink

From what I can ascertain, the backup does not include build artifacts at all. In restoring backups from server A to server B (I always stage a restore of a new version on a second server, then bring it up once I'm happy) I still have to copy the artifacts over from the original server. You should be able to see this simply by looking at the size of the backup file.

Note: your build artifacts should be backed up to a file server independant of your teamcity backup, which is database, config, messages, logs, etc ...

0
Comment actions Permalink

Adam,

Thank you kindly for your reply.  It looks like I'll need to SCP the files in the artifacts folder from the old server to the new server.

Can you tell me about the build-specific files in the /home/teamcity/.BuildServer/config folder?  For each folder in the artifacts folder, there is a matching one in this config folder (although much smaller).

From what I can tell in this document:
http://confluence.jetbrains.net/display/TCD7/TeamCity+Data+Directory

They are:
<project name> — configuration settings specific to a particular project

I'll want to make sure those are on the new server as well.  Do you know if they're included in the webgui backup process?



Another unrelated question, if you would...
Our production server has its URL set to "https://teamcity.domain.com" in the web gui configuration.  I set it that way on my new server "https://teamcity2.domain.com", but it doesn't work.  The new server only responds to "http://teamcity2.domain.com:8111"
I verified the setting is in place on my new server by looking at /home/teamcity/TeamCity/.BuildServer/config/main-config.xml

I compared the /home/teamcity/TeamCity/conf/server.xml file on both machines and they look default.  The old teamcity server still shows connector port 8111.  So I'm not sure how to get the new one to behave like the old one.  Is it something in Tomcat I need to configure?

0
Comment actions Permalink

Hi Ken

The build specific files appear to reside in the build.config (you can find an enourmous amoung of detail here - http://confluence.jetbrains.net/display/TCD7/How+To...#HowTo...-copyserver) , but I've taken a different approach to the upgrade. Given that I needed to do this without interrupting development, the process that I followed was:

1. Set up TeamCity 7.x on the secondary server. Ensure that everything talks to this server via a DNS entry (CNAME) instead of the server itself
2. Create a second (empty) database on the DB server (I'm assuming that given the size of your installation you're running with a db server instead of the internal db).
3. Create a database properties file to point to this one.
4. Restore the complete backup of your 6.5 server to the new database using maintainDB.
5. Copy the system\artifact and system\changes folders across to the new server.
6. Stop the old service and bring the new server up.

This allows you to stage and test an upgrade without affecting your production server (as described in the link above). If everything is working correctly, cut the DNS over from server A to server B and you're up and running - transparently to your users/devs.

As for the tomcat specific configuration for your, I'm not sure. I've always used the installer on windows, but I think it sits in the connector attribute in the conf\server.xml file - perhaps the JetBrains guys have more insight.

0
Comment actions Permalink

Thanks.  It's been a couple days since I looked at the doc you linked.  That helped.

Right now I'm copying over the /system/messages and /system/artifacts folder.
The developers tell me they want those directories.  They're quite large and teamcity documentation keeps mentioning that they are optional.  I wonder how required they actually are.. or what they are.
Our messages directory is 41 gigs and our artifacts folder is about 100 gigs.

Thanks

0
Comment actions Permalink

The messages folders effectively contains all the build logs for all the builds - important information.The artifacts folder contains the output of the builds - essentially all the software that they produce.The link that I gave you is focused around testing a new deployment before you upgrade your production server, so from that perspective they are optional for testing, but certainly not for a final migration. I imagine the devs will be a little miffed if that all disappears.

That said, you may want to ask them about cleanup/archiving of the builds (there is a section in the online help relating to this) - we retain the artifacts for the last 3 builds or last 14 days per project, whichever is higher - everything else is automatically cleaned. When we release software, we pin the build (it's then excluded from the cleanup process). This keeps the sizes manageable.

Where possible, I would urge you to rely on the teamcity backup (and subsequent restore) process instead of attempting to hand-roll your own - this means that the integrity of the system is then maintained by the software itself and not our partial understanding of the product (I'm also just a consumer myself). It also means that as JetBrains adds more features to the system you won't have to manually adjust your processes to keep up.

My suggestion is a daily cron job to trigger a backup of TeamCity via the rest API (see here: http://confluence.jetbrains.net/display/TCD7/TeamCity+Data+Backup) and another to backup the artifacts seperately.

0
Comment actions Permalink

Thanks, that helps.

I'm going to test and document the process of:
1) using the webgui (or scheduled with REST for automation after testing is done) to create a backup to the /.BuildServer/backup folder
2) rsync /system/messages to the new server
3) rsync /system/artifacts to the new server

Hopefully that captures everything.

I see exactly what you're saying about using the builtin mechanism for backups.  I see that our DB had a scheduled cronjob to simply backup the mysql database.  I do not think that is sufficient.

I think we need to "handle" steps 1-3 above on a nightly basis.  But wow, that is a lot of data.

I greatly appreciate your time and insight.

0
Comment actions Permalink

Hmm.  I'm taking a second look.  Are you saying that when I use the webgui to do a backup with full scope, it includes the build logs that are contained in the messages folder?  Does this mean that I do not need to handle the messages folder manually?  Only handle the artifacts folder manually?

I guess I had been misinterpreting the last part of this doc:
http://confluence.jetbrains.net/display/TCD7/TeamCity+Data+Backup

Thanks

0
Comment actions Permalink

Yes, full scope should include all build logs, and you should handle just the artifacts manually.We don't do a full nightly backup of this, just a differential to a network backup location (copying the same data nightly seems a little costly). Then you should obviously copy the teamcity backup to the backup location at the same time.

While performing a full backup of the MySQL database is good, in itself it is insufficient since there is a fair amount of configuration information that is included in the application level backup that is not part of the MySQL database. Additionally, using the maintainDB tool you can restore your backup to a new MySQL instance anyway, so we've opted for only the one level. That's not to say it's bad, just that it's redundant.

Again, if your devs can agree on a cleanup process you can optimize the backup process and reduce the amount of data you store and the time taken to back it up (and obviously restore in a disaster scenario).

0
Comment actions Permalink

I used the webgui to create a full backup on the production teamcity server.
I used scp to copy it over to my new server.

From /home/teamcity/TeamCity/bin I ran
./maintainDB.sh restore -F /home/teamcity/.BuildServer/backup/TeamCity_Backup_20120703_101532.zip -A /home/teamcity/.BuildServer -T /home/teamcity/.BuildServer/config/database.properties

I got the following output with an error:
/home/teamcity/TeamCity/webapps/ROOT/WEB-INF/lib
TeamCity maintenance tool. Copyright 2011 JetBrains s.r.o. All Rights Reserved.

Command line arguments: restore -F /home/teamcity/.BuildServer/backup/TeamCity_Backup_20120703_101532.zip -A /home/teamcity/.BuildServer -T /home/teamcity/.BuildServer/config/database.properties
Using TeamCity data directory: /home/teamcity/.BuildServer
Restoring from backup file: /home/teamcity/.BuildServer/backup/TeamCity_Backup_20120703_101532.zip
Cannot proceed with 'restore' command: Failed to read the metadata version file: central directory is empty, can't expand corrupt archive.
Critical error has occurred during command execution.

I thought that was a bummer.  So I deleted the .zip file on the new teamcity server and scp'd it back over again.  Same error.

Any thoughts on what I could try next or what the problem might be?

0
Comment actions Permalink

The closest thing I could find was this:
http://youtrack.jetbrains.com/issue/TW-18181?query=bug

Its not the same error message.  But I do know that my backup file is 5.4 GB.

0
Comment actions Permalink

Hi Ken

Have you tried creating another backup? I suppose that it's possible that the file got corrupt.

Alternatively, you'll need to log an issue with JetBrains to get it fixed.

0
Comment actions Permalink

Thanks.  I created another backup with the "complete" scope.  This was took considerably less time and was only 1 gig in size.  It looks like someone did some cleaning lately.
I copied it over to the new server and attempted a restore again.
I'm getting an error about the config folder not empty.  Well, no, it isn't empty.  When you install teamcity it puts stuff in there.


[root@teamcity2 config]# /home/teamcity/TeamCity/bin/maintainDB.sh restore -F /home/teamcity/.BuildServer/backup/TeamCity_Backup_20120709_111051.zip -A /home/teamcity/.BuildServer -T /home/teamcity/.BuildServer/config/database.properties
/home/teamcity/TeamCity/webapps/ROOT/WEB-INF/lib
TeamCity maintenance tool. Copyright 2011 JetBrains s.r.o. All Rights Reserved.

Command line arguments: restore -F /home/teamcity/.BuildServer/backup/TeamCity_Backup_20120709_111051.zip -A /home/teamcity/.BuildServer -T /home/teamcity/.BuildServer/config/database.properties
Using TeamCity data directory: /home/teamcity/.BuildServer
Restoring from backup file: /home/teamcity/.BuildServer/backup/TeamCity_Backup_20120709_111051.zip
Backup created by TeamCity version:
        Version: 6.5.3
        Build number: 17985
        Data format version: 454
Cannot proceed with 'restore' command: The configuration directory (/home/teamcity/.BuildServer/config) is not empty.
Critical error has occurred during command execution.
[root@teamcity2 config]#

0
Comment actions Permalink

This topic has not been updated for a long time, but I figured out the "problem" to the error Cannot proceed with 'restore' command: The configuration directory (/home/teamcity/.BuildServer/config) is not empty.

If you want to make a complete overwrite of the backup. You will need to delete the following folders :

C:\ProgramData\JetBrains\TeamCity\config
C:\ProgramData\JetBrains\TeamCity\system

Make a copy of C:\ProgramData\JetBrains\TeamCity\config\database.hsqldb.properties.dist to say C:\ProgramData\JetBrains\TeamCity\database.properties

Then you should be able to run the below command line without much issues.

maintainDB restore -F "path\to\backup.zip" -T "C:\ProgramData\JetBrains\TeamCity\database.properties"

Hope this helps.

0

Please sign in to leave a comment.