Server can't start due to java.lang.OutOfMemoryError: Java heap space Follow
Answered
I've seen similar issues reported, but this one is preventing the server from starting instead of just preventing builds from running. This is a server running on Windows. I tried changing TEAMCITY_SERVER_MEM_OPTS to -Xmx4g but it made no difference. When I attempt to start my server, I get a page that reads as follows:
Unexpected Error
This was not supposed to happen. Please provide the error details to your TeamCity server maintainer.
If you maintain this TeamCity installation and the request seems legit please report this error to JetBrains.
Error: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception: SQL error when doing 'Executing with parameters: [bt17] caused SQL error: ' while performing SQL query: select bs.build_id as build_id, bs.modification_id as mod_id, bs.chain_modification_id as chain_mod_id, bs.branch_name as branch, bs.id as promotion_id, h.is_canceled as canceled, bs.is_personal as personal, bs.is_changes_detached as changes_detached, bs.build_type_id as build_type_id, bs.remove_from_queue_time as remove_from_queue_time, h.status as status, revs.vcs_root_id as rev_root_id, revs.vcs_revision as rev_revision, revs.modification_id as rev_mod_id, revs.checkout_mode as rev_checkout_mode from build_state bs left join build_revisions revs on revs.build_state_id = bs.id left join history h on h.build_id = bs.build_id where bs.build_type_id = ? and bs.is_deleted = 0 and bs.build_id is not null order by bs.build_id : java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space. Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space
Server time: 2020-06-10 09:27:21
TeamCity: 2019.2.3 (build 72031)
Servlet container: Apache Tomcat/8.5.47
Trace: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception: SQL error when doing 'Executing with parameters: [bt17] caused SQL error: ' while performing SQL query: select bs.build_id as build_id, bs.modification_id as mod_id, bs.chain_modification_id as chain_mod_id, bs.branch_name as branch, bs.id as promotion_id, h.is_canceled as canceled, bs.is_personal as personal, bs.is_changes_detached as changes_detached, bs.build_type_id as build_type_id, bs.remove_from_queue_time as remove_from_queue_time, h.status as status, revs.vcs_root_id as rev_root_id, revs.vcs_revision as rev_revision, revs.modification_id as rev_mod_id, revs.checkout_mode as rev_checkout_mode from build_state bs left join build_revisions revs on revs.build_state_id = bs.id left join history h on h.build_id = bs.build_id where bs.build_type_id = ? and bs.is_deleted = 0 and bs.build_id is not null order by bs.build_id : java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space. Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space at org.hsqldb.jdbc.JDBCUtil.sqlException(JDBCUtil.java:418) at org.hsqldb.jdbc.JDBCUtil.sqlException(JDBCUtil.java:247) at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(JDBCPreparedStatement.java:4655) at org.hsqldb.jdbc.JDBCPreparedStatement.execute(JDBCPreparedStatement.java:1098) at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:191) at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:191) at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:191) at jetbrains.buildServer.serverSide.db.queries.GenericQuery$8.run(GenericQuery.java:356) at jetbrains.buildServer.serverSide.db.queries.GenericQuery$8.run(GenericQuery.java:353) at jetbrains.buildServer.serverSide.db.DBInternalUtils.rawOperation(DBInternalUtils.java:70) at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:353) at jetbrains.buildServer.serverSide.db.queries.GenericQuery.access$500(GenericQuery.java:27) at jetbrains.buildServer.serverSide.db.queries.GenericQuery$7.action(GenericQuery.java:339) at jetbrains.buildServer.serverSide.db.queries.GenericQuery.underPreparedStatement(GenericQuery.java:409) at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:336) at jetbrains.buildServer.serverSide.db.queries.GenericQuery.execute(GenericQuery.java:78) at jetbrains.buildServer.serverSide.impl.BuildTypeOrderedBuildsImpl.lambda$loadBuilds$4(BuildTypeOrderedBuildsImpl.java:108) at jetbrains.buildServer.serverSide.db.InternalDBUtil.runAndRetry(InternalDBUtil.java:25) at jetbrains.buildServer.serverSide.db.TeamCityDatabaseManager.runAndRetry(TeamCityDatabaseManager.java:1142) at jetbrains.buildServer.serverSide.db.DBFacade.runAndRetry(DBFacade.java:292) at jetbrains.buildServer.serverSide.impl.BuildTypeOrderedBuildsImpl.loadBuilds(BuildTypeOrderedBuildsImpl.java:143) at jetbrains.buildServer.serverSide.impl.BuildTypeOrderedBuildsImpl.getBuilds(BuildTypeOrderedBuildsImpl.java:71) at jetbrains.buildServer.serverSide.impl.BuildTypeOrderedBuildsImpl.consumeBuilds(BuildTypeOrderedBuildsImpl.java:35) at jetbrains.buildServer.serverSide.impl.BuildTypeBranches.getRecentlyUsedBuildBranches(BuildTypeBranches.java:151) at jetbrains.buildServer.serverSide.impl.BuildTypeBranches.access$200(BuildTypeBranches.java:262) at jetbrains.buildServer.serverSide.impl.BuildTypeBranches$ActiveBuildBranches.readFromDb(BuildTypeBranches.java:40) at jetbrains.buildServer.serverSide.impl.BuildTypeBranches$ActiveBuildBranches.getValue(BuildTypeBranches.java:38) at jetbrains.buildServer.serverSide.impl.BuildTypeBranches.getBranchInfo(BuildTypeBranches.java:212) at jetbrains.buildServer.serverSide.impl.BuildTypeBranches.getBranches(BuildTypeBranches.java:46) at jetbrains.buildServer.serverSide.impl.BuildTypeImpl.getBranches(BuildTypeImpl.java:508) at jetbrains.buildServer.controllers.project.ProjectWebUtil.lambda$projectBranchesAnyMatch$2(ProjectWebUtil.java:27) at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:516) at jetbrains.buildServer.controllers.project.ProjectWebUtil.projectBranchesAnyMatch(ProjectWebUtil.java:10) at jetbrains.buildServer.controllers.project.ProjectWebUtil.getBranchBeansForProjects(ProjectWebUtil.java:43) at jetbrains.buildServer.controllers.overview.OverviewBean.<init>(OverviewBean.java:85) at jetbrains.buildServer.controllers.overview.OverviewController.getOverviewBean(OverviewController.java:28) at jetbrains.buildServer.controllers.overview.OverviewController.doHandle(OverviewController.java:47) at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:114) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174) at jetbrains.buildServer.controllers.BaseController.handleRequest(BaseController.java:93) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:1) at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:14) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:104) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:5) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at jetbrains.buildServer.web.UserIdProviderFilter.doFilter(UserIdProviderFilter.java:12) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at jetbrains.buildServer.web.NodeInfoHeaderFilter.doFilter(NodeInfoHeaderFilter.java:9) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107) at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:54) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112) at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:2) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112) at jetbrains.buildServer.diagnostic.web.HttpRequestsDurationMetricsReporter.doFilter(HttpRequestsDurationMetricsReporter.java:9) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112) at jetbrains.buildServer.web.HttpSecurityHeadersFilter.doFilter(HttpSecurityHeadersFilter.java:62) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112) at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73) at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:61) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:43) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space ... 108 more Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError: Java heap space at org.hsqldb.error.Error.error(Error.java:113) at org.hsqldb.persist.DataFileCache.getFromFile(DataFileCache.java:1078) at org.hsqldb.persist.DataFileCache.get(DataFileCache.java:1042) at org.hsqldb.persist.RowStoreAVLDisk.get(RowStoreAVLDisk.java:118) at org.hsqldb.index.NodeAVLDisk.findNode(NodeAVLDisk.java:186) at org.hsqldb.index.NodeAVLDisk.getRight(NodeAVLDisk.java:250) at org.hsqldb.index.IndexAVL.findNode(IndexAVL.java:1742) at org.hsqldb.index.IndexAVL.findFirstRow(IndexAVL.java:1171) at org.hsqldb.RangeVariable$RangeIteratorMain.getFirstRow(RangeVariable.java:1356) at org.hsqldb.RangeVariable$RangeIteratorMain.initialiseIterator(RangeVariable.java:1265) at org.hsqldb.RangeVariable$RangeIteratorMain.next(RangeVariable.java:1203) at org.hsqldb.QuerySpecification.buildResult(QuerySpecification.java:1533) at org.hsqldb.QuerySpecification.getSingleResult(QuerySpecification.java:1439) at org.hsqldb.QuerySpecification.getResult(QuerySpecification.java:1429) at org.hsqldb.StatementQuery.getResult(StatementQuery.java:66) at org.hsqldb.StatementDMQL.execute(StatementDMQL.java:195) at org.hsqldb.Session.executeCompiledStatement(Session.java:1378) at org.hsqldb.Session.execute(Session.java:991) at org.hsqldb.jdbc.JDBCPreparedStatement.fetchResult(JDBCPreparedStatement.java:4647) ... 105 more Caused by: java.lang.OutOfMemoryError: Java heap space
Please sign in to leave a comment.
Hi,
thanks for your report. The "Java heap space" error typically indicates that the server does not have enough memory to run. If you added the TEAMCITY_SERVER_MEM_OPTS, there are two points to keep in mind:
-This environment variable has to be picked up by the process starting up. If you run the server as a service, you might need to adapt the user that runs it, restart the server, etc. During the startup process, the server logs the "JVM parameters". You can find this line and see if the -Xmx is being applied or not. You can also test by stopping the service (if applies), opening a command line, exporting the variable there and from the same shell spawning the server, which should then inherit it.
-Any amount above 2Gb in theory, around 1 - 1.2Gb in practice, will only work if you are using a 64 bit Java, since 32 bit cannot manage so much memory. If your installation is using 32 bit java it will not work, fail with the same issue, but require changing to 64 to work. We have instructions for it here: https://www.jetbrains.com/help/teamcity/installing-and-configuring-the-teamcity-server.html#InstallingandConfiguringtheTeamCityServer-Using64bitJavatoRunTeamCityServer
Furthermore, the HSQLDB system is not recommended for production servers. If you are still using the integrated database, please consider migrating it to an external database: https://www.jetbrains.com/help/teamcity/migrating-to-an-external-database.html