org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space

For some reason Teamcity went down on us and we are receivingthe following exceptions.


Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [jetbrains.buildServer.serverSide.impl.AgentPersistenceFacade]: Constructor threw exception; nested exception is jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception SQLException/HsqlException: SQL error when doing: DML query
SQL query: update agent set registered=0, unregistered_reason=? where registered=1
SQL exception: java.lang.OutOfMemoryError: Java heap space
 at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:164)
 at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:125)
 at jetbrains.buildServer.spring.InstantiationStrategySelector$1.instantiate(InstantiationStrategySelector.java:61)
 at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:270)
 ... 151 more
Caused by: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception SQLException/HsqlException: SQL error when doing: DML query
SQL query: update agent set registered=0, unregistered_reason=? where registered=1
SQL exception: java.lang.OutOfMemoryError: Java heap space
 at org.hsqldb.jdbc.JDBCUtil.sqlException(JDBCUtil.java:418)
 at org.hsqldb.jdbc.JDBCUtil.sqlException(JDBCUtil.java:247)
 at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(JDBCPreparedStatement.java:4655)
 at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(JDBCPreparedStatement.java:310)
 at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
 at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
 at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
 at jetbrains.buildServer.serverSide.db.DBFunctions$9.run(DBFunctions.java:921)
 at jetbrains.buildServer.serverSide.db.DBFunctions$9.run(DBFunctions.java:918)
 at jetbrains.buildServer.serverSide.db.DBInternalUtils.rawOperation(DBInternalUtils.java:72)
 at jetbrains.buildServer.serverSide.db.DBFunctions.executeDmlInternal(DBFunctions.java:917)
 at jetbrains.buildServer.serverSide.db.DBFunctions.executeDml(DBFunctions.java:875)
 at jetbrains.buildServer.serverSide.db.queries.GenericQuery$3.run(GenericQuery.java:139)
 at jetbrains.buildServer.serverSide.db.queries.GenericQuery$3.run(GenericQuery.java:137)
 at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:234)
 at jetbrains.buildServer.serverSide.db.queries.GenericQuery.executeUpdate(GenericQuery.java:137)
 at jetbrains.buildServer.serverSide.impl.AgentPersistenceFacade.<init>(AgentPersistenceFacade.java:264)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:148)
 ... 154 more
Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
 ... 176 more
Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
 at org.hsqldb.error.Error.error(Error.java:113)
 at org.hsqldb.persist.DataFileCache.getFromFile(DataFileCache.java:1078)
 at org.hsqldb.persist.DataFileCache.get(DataFileCache.java:1042)
 at org.hsqldb.persist.RowStoreAVLDisk.get(RowStoreAVLDisk.java:118)
 at org.hsqldb.index.NodeAVLDisk.findNode(NodeAVLDisk.java:186)
 at org.hsqldb.index.NodeAVLDisk.getRight(NodeAVLDisk.java:250)
 at org.hsqldb.index.IndexAVL.next(IndexAVL.java:1436)
 at org.hsqldb.index.IndexAVL.next(IndexAVL.java:1374)
 at org.hsqldb.index.IndexAVL$IndexRowIterator.getNextRow(IndexAVL.java:1999)
 at org.hsqldb.RangeVariable$RangeIteratorMain.findNext(RangeVariable.java:1372)
 at org.hsqldb.RangeVariable$RangeIteratorMain.next(RangeVariable.java:1206)
 at org.hsqldb.StatementDML.executeUpdateStatement(StatementDML.java:510)
 at org.hsqldb.StatementDML.getResult(StatementDML.java:197)
 at org.hsqldb.StatementDMQL.execute(StatementDMQL.java:195)
 at org.hsqldb.Session.executeCompiledStatement(Session.java:1378)
 at org.hsqldb.Session.execute(Session.java:991)
 at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(JDBCPreparedStatement.java:4647)
 ... 173 more
Caused by: java.lang.OutOfMemoryError: Java heap space
 at org.hsqldb.lib.StringConverter.readUTF(StringConverter.java:477)
 at org.hsqldb.rowio.RowInputBinary.readString(RowInputBinary.java:92)
 at org.hsqldb.rowio.RowInputBinary.readChar(RowInputBinary.java:109)
 at org.hsqldb.rowio.RowInputBase.readData(RowInputBase.java:180)
 at org.hsqldb.rowio.RowInputBinary.readData(RowInputBinary.java:65)
 at org.hsqldb.rowio.RowInputBase.readData(RowInputBase.java:159)
 at org.hsqldb.rowio.RowInputBinary.readData(RowInputBinary.java:253)
 at org.hsqldb.rowio.RowInputBinaryDecode.readData(RowInputBinaryDecode.java:66)
 at org.hsqldb.RowAVLDisk.<init>(RowAVLDisk.java:163)
 at org.hsqldb.persist.RowStoreAVLDisk.get(RowStoreAVLDisk.java:165)
 at org.hsqldb.persist.DataFileCache.getFromFile(DataFileCache.java:1067)
 at org.hsqldb.persist.DataFileCache.get(DataFileCache.java:1042)
 at org.hsqldb.persist.RowStoreAVLDisk.get(RowStoreAVLDisk.java:118)
 at org.hsqldb.index.NodeAVLDisk.findNode(NodeAVLDisk.java:186)
 at org.hsqldb.index.NodeAVLDisk.getRight(NodeAVLDisk.java:250)
 at org.hsqldb.index.IndexAVL.next(IndexAVL.java:1436)
 at org.hsqldb.index.IndexAVL.next(IndexAVL.java:1374)
 at org.hsqldb.index.IndexAVL$IndexRowIterator.getNextRow(IndexAVL.java:1999)
 at org.hsqldb.RangeVariable$RangeIteratorMain.findNext(RangeVariable.java:1372)
 at org.hsqldb.RangeVariable$RangeIteratorMain.next(RangeVariable.java:1206)
 at org.hsqldb.StatementDML.executeUpdateStatement(StatementDML.java:510)
 at org.hsqldb.StatementDML.getResult(StatementDML.java:197)
 at org.hsqldb.StatementDMQL.execute(StatementDMQL.java:195)
 at org.hsqldb.Session.executeCompiledStatement(Session.java:1378)
 at org.hsqldb.Session.execute(Session.java:991)
 at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(JDBCPreparedStatement.java:4647)
 at org.hsqldb.jdbc.JDBCPreparedStatement.executeUpdate(JDBCPreparedStatement.java:310)
 at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
 at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
 at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
 at jetbrains.buildServer.serverSide.db.DBFunctions$9.run(DBFunctions.java:921)
 at jetbrains.buildServer.serverSide.db.DBFunctions$9.run(DBFunctions.java:918)
[2015-11-26 09:44:11,997]   INFO -  jetbrains.buildServer.STARTUP - Current stage: TeamCity server startup error
[2015-11-26 09:44:11,997]   INFO -  jetbrains.buildServer.STARTUP - Administrator can login from web UI using authentication token: 2213626179418305609




I tired running the Database Manager found in

https://devnet.jetbrains.com/message/5443181?tstart=0


expectnig it to error but loads up fine. Has anyone else seen this?






1 comment

Hi Edward,

What TeamCity version do you use? There is a related issue https://youtrack.jetbrains.com/issue/TW-40425, that is already fixed. As a workaround, please stop the server, make sure there are no java processes running, then navigate to <TeamCity data directory>\system\caches\buildsMetadata and remove contents of the directory and try starting server again.

Also please consider migrating to the external database. We strongly recommend using an external database as a back-end TeamCity database in a production environment.

0

Please sign in to leave a comment.