157 out of memory

We ran out of memory on the vmmachine that team city is on, we have now double the memory and re-started service and machine but still get error as below:

Any other suggestions welcome

Ooops, something went wrong...

Error on step: Waiting for a database server...

SQL error when doing: Connecting to HSQLDB
SQL exception: error in script file line: 157 out of memory


Copy to clipboard

Stacktrace copied!

jetbrains.buildServer.serverSide.db.UnexpectedDBException: SQL error when doing: Connecting to HSQLDB

SQL exception: error in script file line: 157 out of memory

   at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1225)

   at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)

   at jetbrains.buildServer.serverSide.db.TeamCityDataSource.checkDatabaseConnection(TeamCityDataSource.java:24)

   at jetbrains.buildServer.serverSide.db.TeamCityDataSource.connect(TeamCityDataSource.java:9)

   at  jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.processStartupServerInternal(TeamCityDispatcherServlet.java:22)

   at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.processServerStartup(TeamCityDispatcherServlet.java:84)

   at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.access$000(TeamCityDispatcherServlet.java:96)

   at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet$1.run(TeamCityDispatcherServlet.java:1)

   at java.lang.Thread.run(Thread.java:595)

caused by: java.sql.SQLException: error in script file line: 157 out of memory

   at org.hsqldb.jdbc.Util.sqlException(Unknown Source)

   at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)

   at org.hsqldb.jdbcDriver.getConnection(Unknown Source)

   at org.hsqldb.jdbcDriver.connect(Unknown Source)

   at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)

   at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)

   at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1247)

   at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1221)

   at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)

   at jetbrains.buildServer.serverSide.db.TeamCityDataSource.checkDatabaseConnection(TeamCityDataSource.java:24)

   at jetbrains.buildServer.serverSide.db.TeamCityDataSource.connect(TeamCityDataSource.java:9)

   at  jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.processStartupServerInternal(TeamCityDispatcherServlet.java:22)

   at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.processServerStartup(TeamCityDispatcherServlet.java:84)

   at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.access$000(TeamCityDispatcherServlet.java:96)

   at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet$1.run(TeamCityDispatcherServlet.java:1)

   at java.lang.Thread.run(Thread.java:595)
10 comments

Hi, Tara!

It looks like HSQL data files are corrupted. How much memory you've specified in -Xmx parameter?
Which version and build number of TeamCity software?

Thanks,
Leonid

0

We are using Team City 5.0.2.  We set the memory of the VMMachine to 512mb, where can I find the Xmx parameter?

0

Hello Tara,

The 512M for a virtual machine TeamCity running in is too small amount. I can suggest you to increase VM memory up to 1.5-2 Gb.

As of the Xmx parameter, please see details in our TeamCity documentation:
http://confluence.jetbrains.net/display/TCD5/Installing+and+Configuring+the+TeamCity+Server#InstallingandConfiguringtheTeamCityServer-SettingUpMemorysettingsforTeamCityServer

Please let me know if the problem still exists.

Thanks,
Leonid

0

We run tomcat as windows service so found following for updating the java properties

TeamCity Server is Run as Windows Service

To edit JVM server options run Tomcat's service configuration editor by executing the command

tomcat6w.exe //ES//TeamCity

in <TeamCity home>\bin directory and then edit the Java Options on the Java tab (for more information see Tomcat 6 documentation).

To change heap memory dedicated to the JVM change the "Maximum memory pool" setting.

What would you recommend setting these too?  We are going to set memory of vm to 1.5g.  Thanks, Tara.  I take it I then stop and start windows service for tomcat?

--JvmMs Initial memory pool size in MB
--JvmMx Maximum memory pool size in MB
--JvmSs

Thread stack size in KB

0

> What would you recommend setting these too?  We are going to set memory of vm to 1.5g.

What do you have there now? I'd set it to 750 and restart the service afterwards.

Anyway it looks like the database become corrupted and in such case the easiest recovery is to restore latest good backup (if you have one) and then migrate to external database as we do not recommend using internal (HSQLDB) database exactly because it can become corrupted.

0

Hi

I've tried setting maximum memory pool to 750mb and XX:MaxPermSize to 150m but and restarted service but it has made no difference.  Is this right?  Could the HSQLDB be corrupt?  If I re-create will I still have the build configurations?

Thanks
Tara

0

Unfortunately we don't have backup for this and was not involved in setting this up.  I found this for re-created the HSQLDB:


  Most likely your HSQLDB database got corrupted :(. You need either to re-create it or (better) setup MysqlDB: http://www.jetbrains.net/confluence/display/TCD/MigratingtoMySQL+database



  If you want to recreate HSQLDB database, follow the following steps:

  1. Backup your artifacts from .BuildServer/system/artifacts directory, if needed

  2. Remove .BuildServer/system/[artifacts|messages|changes|buildserver.*] files and directories

  3. Restart TeamCity. You configurations will remain, but you'll have to recreate user accounts and build history will be empty.

If I create a Mysql database, will we have to set up build configurations again?  Or can I do as advise above so I keep them and the migrate?

Thanks
Tara

0

Hi

I successfuly create a SQL 2005 database for team city and managed to keep the build configurations so I'm no longer using HSQLDB as recommended.  I followed notes (http://confluence.jetbrains.net/display/TCD4/Setting+up+an+External+Database) which were useful.

I now need to do the same for another team city server i.e. increase the memory of vm machine and set up a SQL 2005 database, however this time I would like to migrate the data as this is not corrupted.  I have been following the notes (http://confluence.jetbrains.net/display/TCD4/Migrating+to+an+External+Database) but when I get to step 5. below, the file dbMigration.properties does not exist, are you able to provide this file?

5.  Edit the <TeamCity home>/bin/dbMigration.properties file,  entering correct values for the source and target database. Ensure all  the properties are supplied and only the necessary lines are uncommented  (single-line comments are designated by # character)

Thanks
Tara

0

I also don't appear to have the migrateDB script either?

Thanks in advance

0

As you seems to be using TeamCity 5.0, please refer to documentation for TeamCity 5.0.

0

Please sign in to leave a comment.