TeamCity stuck Connecting to the database after Windows Server Update reboot

Unfortunately automatic updates was enabled and rebooted our teamcity server last night. Now, whenever we try to access the server via browser we get "TeamCity Maintenance: Connecting to the database". I looked at the teamcity logs and sure enough that's the last thing that's logged. We haven't had the time to migrate to an external database so it's still using the internal HSQL DB. Attached is a portion of our logs starting from just before the windows updates last night. The reboot was initiated at 3:08:48 AM. At 3AM our box executes cleanup but that was done 7 minutes before the restart was triggered. After looking through the teamcity log and Windows Event Viewer it doesn't look like windows signaled the TeamCity Server service to shutdown but it DID signal the build agent service, running on the same box, to shutdown.

Other possibly helpful information:
TeamCity 8.0.1 (build 27435)
Windows Server 2008 R2

I need this up and running ASAP and have not run into issues like this before.
Let me know what other information or logs could be useful.

Thanks

Update:
I'd also like to add that the buildserver.log file in E:\ProgramData\JetBrains\TeamCity\system has not been modified since right before the reboot last night @ 3:08AM while buildserver.backup and buildserver.data have.
I rebooted the server twice but the TeamCity service still cannot connect to the database. The java process for the service appears to have one thread using 15% CPU (100% for 1 of 8 Cores).

This is the only update that was installed last night http://support.microsoft.com/kb/2858725/en-us



Attachment(s):
teamcity-server.log.zip
6 comments
Comment actions Permalink

Hello Justin,

the internal (HSQL) database is very unstable and should not be used in production (because it may crash in any moment).

Now, the best way is to restore database from your backup.
See instructions how to do it: http://confluence.jetbrains.com/display/TCD8/Restoring+TeamCity+Data+from+Backup

Best regards,
Leonid


P.S. Don't forget to migrate to an external database when TC is restored — HSQL may crash again.

0
Comment actions Permalink

How large should the backup file be compared to the data file? My server's backup file is a little over 5MB versus 409MB for the data file. I just want to make sure that's not unusual before I attempt this.

0
Comment actions Permalink

> How large should the backup file be compared to the data file?
Do you mean TeamCity backup file or HSQL backup file?

In case of TeamCity one — it depeneds on your database size.
In case of HSQL one — I don't know.

If you didn't do TeamCity backup, you can recover TeamCity by removing <DataDir>/system/buildserver.* files. However, all build history, tests' statuses, user settings will be lost. TeamCity server will create a new database on the next start.

0
Comment actions Permalink

Thanks Leonid.

What about build configurations? Are those part of user settings and would they be lost if I delete <DataDir>/system/buildserver.* files? I have only been using TC for about 6 or 7 months and haven't had to worry about where things are stored up until now.

0
Comment actions Permalink

> What about build configurations?
They're stored in the Data Directory and they'll be preserved when HSQL DB files have dropped.

And, I'd recommend migrating to an external database (PostgreSQL, for example), and not forget to start backup time to time.

Currently, TC has no automatic scheduled backup (you're welcome to watch/vote for http://youtrack.jetbrains.com/issue/TW-6448 - I'm planning to implement it in 9.x).

0
Comment actions Permalink

Leonid, I just wanted to thank you for your help and update you on my progress. I was able to get everything back up and running last week and copied off my "lost" artifacts from previous releases. Just as you said I still had my build configurations so that made me very happy. This week we'll be migrating to an external DB as you suggested so hopefully we can avoid this problem going forward. Thanks again for your help.

Justin

0

Please sign in to leave a comment.