TeamCity seems stuck after maintenance .

Hi,

We are still evaluating TeamCity with two projects and 9 configurations. We encounter a problem when the server runs the database maintenance process. The server seems stuck and it's not possible to access any page on the server. This behavior is encountered everyday.
We get a page with "Server is maintaining its database at the moment. It may take from several minutes to several hours depending on the size of the database."

The logs in teamcity-server.log are:

INFO - jetbrains.buildServer.SERVER - Compacting database...
INFO - rver.serverSide.VersionChecker - Checking for new version...
INFO - rver.serverSide.VersionChecker - New version is not available
INFO - pl.cleanup.HistoryEntryCleaner - Deleting messages file C:\DEV\.BuildServer\system\messages\CH64\64.build
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@f3a5a5
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@14f3
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@15f7c4b
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@1d93525
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@1302ed5
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@52bb2d
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@17eb194
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@bc86fc
INFO - rver.serverSide.VersionChecker - Checking for new version...
INFO - rver.serverSide.VersionChecker - New version is not available
INFO - pl.cleanup.HistoryEntryCleaner - Deleting messages file C:\DEV\.BuildServer\system\messages\CH65\65.build
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@f3a5a5
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@14f3
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@15f7c4b
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@1d93525
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@1302ed5
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@52bb2d
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@17eb194
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@bc86fc
INFO - rver.serverSide.VersionChecker - Checking for new version...
INFO - rver.serverSide.VersionChecker - New version is not available
INFO - pl.cleanup.HistoryEntryCleaner - Deleting messages file C:\DEV\.BuildServer\system\messages\CH66\66.build
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@f3a5a5
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@14f3
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@15f7c4b
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@1d93525
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@1302ed5
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@52bb2d
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@17eb194
INFO - pl.cleanup.HistoryEntryCleaner - Executing org.apache.commons.dbcp.DelegatingPreparedStatement@bc86fc

The database is currently the embedded one and has as size of 370MB

We have to restart the Teamcity process to be able to work again.

What do you think?

13 comments

Hi,

Currently, TeamCity requires database maintenance period to cleanup obsoleted data (like old builds).
Usually, this takes several minutes for configurations like yours (but depends on number of builds you create daily).
You can specify the exact time when maintenance start, and the policy for the cleanup. See "Build History Clean-up" section on the Administration page.

We also plan to remove the need for dedicated cleanup time. You may watch/vote for
http://www.jetbrains.net/jira/browse/TW-2527

Thanks for trying TeamCity,
KIR

0

I'm aware about that.
But the clean up process is working for hours.
This morning I restarted the "teamcity" process, while it was displaying that the cleanup process was running for 5:32. The server is running on a bi-opteron processor with 4GB of memory.
It seems strange to me that it takes so long (The database is around 380MB)

0

Hello,

Do you have build configurations with inspection or duplicate runners?

--
Pavel Sher

"Regis Le Brettevillois" <no_reply@jetbrains.com> wrote in message
news:15131193.1181735730121.JavaMail.itn@is.intellij.net...

I'm aware about that.
But the clean up process is working for hours.
This morning I restarted the "teamcity" process, while it was displaying
that the cleanup process was running for 5:32. The server is running on a
bi-opteron processor with 4GB of memory.
It seems strange to me that it takes so long (The database is around
380MB)



0

Yes, we have inspection and duplicate runners.
We discovered that the problem is related to a OutOfMemory error, but we increased the memory up to 1 GB, we still have the server freezing.

Regis

0

What database do you use? If you are using HSQLDB (this database is used by
default) then I would suggest you to switch your database to MySQL. I think
this should solve your problems with cleanup. You can read more about data
migration here:
http://www.jetbrains.net/confluence/display/TCD/MigratingtoMySQL+database

--
Pavel Sher

"Regis Le Brettevillois" <no_reply@jetbrains.com> wrote in message
news:29255919.1182170303723.JavaMail.itn@is.intellij.net...

Yes, we have inspection and duplicate runners.
We discovered that the problem is related to a OutOfMemory error, but we
increased the memory up to 1 GB, we still have the server freezing.

>

Regis



0

Hello Regis,

I've created an issue to address the problem you experience: http://www.jetbrains.net/jira/browse/TW-2846

--
Best regards,

Yegor Yarko
Quality Assurance Engineer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Regis,

Concerning migration to MySQL:
if your server runs under Windows, you can try to use database migration .bat file attached to the issue http://www.jetbrains.net/jira/browse/TW-2518

--
Best regards,

Yegor Yarko
Quality Assurance Engineer
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Hi TeamCity team,

Thanks for your helpful information. I try to convert the HSQLDB database to MySQL to solve the problem we encounter.

But I face a new problem during conversion:

Tue Jun 19 10:15:52 CEST 2007 Initializing schema...
TABLES to be converted: (31) [[agent, agent_sources_version, build_queue, cancel
ed_info, compiler_output, deleted_build_types, downloaded_artifacts, failed_test
s_output, granted_license, history, ignored_tests, old_sources_on_agent, paused_
build_types, personal_vcs_changes, personal_vcs_history, remember_me, remote_run
buildtypes, running, server, server_property, statistics, test_info, user, use
r_blocks, user_notification_events, user_projects_order, user_projects_visibilit
y, user_property, user_watch_type, vcs_changes, vcs_history]]
Tue Jun 19 10:15:56 CEST 2007 Exporting data to TC-export-1182240956011
Exception in thread "main" org.dbunit.dataset.DataSetException: java.sql.SQLExce
ption: S1000 General error java.util.NoSuchElementException
at org.dbunit.database.DatabaseDataSet.getTable(DatabaseDataSet.java:234
)
at org.dbunit.database.DatabaseTableIterator.getTable(DatabaseTableItera
tor.java:70)
at org.dbunit.dataset.filter.AbstractTableFilter$FilterIterator.getTable
(AbstractTableFilter.java:111)
at org.dbunit.dataset.stream.DataSetProducerAdapter.produce(DataSetProdu
cerAdapter.java:61)
at org.dbunit.dataset.csv.CsvDataSetWriter.write(CsvDataSetWriter.java:7
7)
at org.dbunit.dataset.csv.CsvDataSetWriter.write(CsvDataSetWriter.java:2
11)
at jetbrains.buildServer.serverSide.HSQL2MySQL.doExport(HSQL2MySQL.java:
57)
at jetbrains.buildServer.serverSide.HSQL2MySQL.main(HSQL2MySQL.java:84)
java.sql.SQLException: S1000 General error java.util.NoSuchElementException
at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source)
at org.hsqldb.jdbc.jdbcStatement.executeQuery(Unknown Source)
at org.dbunit.database.AbstractResultSetTable.(AbstractResultSetTa ble.java:91) at org.dbunit.database.ForwardOnlyResultSetTable.]]>(ForwardOnlyResul
tSetTable.java:50)
at org.dbunit.database.ForwardOnlyResultSetTableFactory.createTable(Forw
ardOnlyResultSetTableFactory.java:45)
at org.dbunit.database.DatabaseDataSet.getTable(DatabaseDataSet.java:230
)
at org.dbunit.database.DatabaseTableIterator.getTable(DatabaseTableItera
tor.java:70)
at org.dbunit.dataset.filter.AbstractTableFilter$FilterIterator.getTable
(AbstractTableFilter.java:111)
at org.dbunit.dataset.stream.DataSetProducerAdapter.produce(DataSetProdu
cerAdapter.java:61)
at org.dbunit.dataset.csv.CsvDataSetWriter.write(CsvDataSetWriter.java:7
7)
at org.dbunit.dataset.csv.CsvDataSetWriter.write(CsvDataSetWriter.java:2
11)
at jetbrains.buildServer.serverSide.HSQL2MySQL.doExport(HSQL2MySQL.java:
57)
at jetbrains.buildServer.serverSide.HSQL2MySQL.main(HSQL2MySQL.java:84)
Deleting temporary files...

The temporary repository contains these files just after the exception:

755 AGENT.csv
1 595 AGENT_SOURCES_VERSION.csv
54 BUILD_QUEUE.csv
469 CANCELED_INFO.csv
736 655 COMPILER_OUTPUT.csv
15 DELETED_BUILD_TYPES.csv
8 746 DOWNLOADED_ARTIFACTS.csv

What do you think?

0

Regis Le Brettevillois wrote:

This really looks like HSQL database is corrupted, this may be also
reason of original "stuck" issue.

One possible solution I see is that I will write special version of the
migration tool which will try to export as much data as possible. Than
we can try to recover some consistent state to import into mysql.

Hi TeamCity team,

Thanks for your helpful information. I try to convert the HSQLDB database to MySQL to solve the problem we encounter.

But I face a new problem during conversion:

Tue Jun 19 10:15:52 CEST 2007 Initializing schema...
TABLES to be converted: (31) [[agent, agent_sources_version, build_queue, cancel
ed_info, compiler_output, deleted_build_types, downloaded_artifacts, failed_test
s_output, granted_license, history, ignored_tests, old_sources_on_agent, paused_
build_types, personal_vcs_changes, personal_vcs_history, remember_me, remote_run
buildtypes, running, server, server_property, statistics, test_info, user, use
r_blocks, user_notification_events, user_projects_order, user_projects_visibilit
y, user_property, user_watch_type, vcs_changes, vcs_history]]
Tue Jun 19 10:15:56 CEST 2007 Exporting data to TC-export-1182240956011
Exception in thread "main" org.dbunit.dataset.DataSetException: java.sql.SQLExce
ption: S1000 General error java.util.NoSuchElementException
at org.dbunit.database.DatabaseDataSet.getTable(DatabaseDataSet.java:234
)

Deleting temporary files...

The temporary repository contains these files just after the exception:

755 AGENT.csv
1 595 AGENT_SOURCES_VERSION.csv
54 BUILD_QUEUE.csv
469 CANCELED_INFO.csv
736 655 COMPILER_OUTPUT.csv
15 DELETED_BUILD_TYPES.csv
8 746 DOWNLOADED_ARTIFACTS.csv

What do you think?



--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0

We try to migrate the database manually by copying table content from HSQLDB to MySQL (using copy tool from Squirrel SQL).
What are the tables that don't have to be copied?

Thanks

0

For information the table that is corrupted is TEST_INFO, all others seem ok.

0

Sorry for a delay, the news gate is lagging.

W/o test_info and failed_test_output you will loose failed/ignored test info from your history, but general build status and description will be retained.

Try to copy as many data as possible and post list of tables you was unable to save - I will help you maintain as much consistency as possible.

0

Thank you for your feedback.
We migrated the whole database except agent and server related tables.
The only corrupted table is TEST_INFO.
We restarted the server on the migrated MySQL database and it seems stable.
A clean-up pass seems to have been successful with no more frozen server.

0

Please sign in to leave a comment.