3776 build Sql Exception

After upgrading today to 3776 we got this exception 2 times crashing the TeamCity server when 6-9 builds were queued. We are using Maven.

ERROR - verSide.impl.XmlRpcBasedServer - Trouble logging BuildMessage1 BlockEnd $TEST_BLOCK$ testDeleteItemList DefaultMessage

org.springframework.jdbc.UncategorizedSQLException: ConnectionCallback; uncategorized SQLException for SQL []; SQL state ; error code ; S1000 General error java.lang.NullPointerException in statement ; nested exception is java.sql.SQLException: S1000 General error java.lang.NullPointerException in statement
java.sql.SQLException: S1000 General error java.lang.NullPointerException in statement
at org.hsqldb.jdbc.Util.throwError(Unknown Source)
at org.hsqldb.jdbc.jdbcPreparedStatement.execute(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:168)
at jetbrains.buildServer.serverSide.impl.query.Query.execute(Query.java:27)
at jetbrains.buildServer.serverSide.impl.query.Query$3.run(Query.java)
at jetbrains.buildServer.serverSide.impl.DBFacade$13.doInConnection(DBFacade.java)
at jetbrains.buildServer.serverSide.impl.DBFacade$15.doInConnection(DBFacade.java)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:230)
at jetbrains.buildServer.serverSide.impl.DBFacade._runSql(DBFacade.java:219)
at jetbrains.buildServer.serverSide.impl.DBFacade.runSql(DBFacade.java:186)
at jetbrains.buildServer.serverSide.impl.DBFacade.runSql(DBFacade.java:10)
at jetbrains.buildServer.serverSide.impl.query.Query.execute(Query.java:11)
at jetbrains.buildServer.serverSide.impl.query.UpdateFailedTestOutputQuery.execute(UpdateFailedTestOutputQuery.java:3)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor$2.processBlockEnd(AntMessagesProcessor.java:6)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor.processBlockEnd(AntMessagesProcessor.java:103)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor.access$1200(AntMessagesProcessor.java:90)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor$6.processMessage(AntMessagesProcessor.java:1)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor.doProcessMessage(AntMessagesProcessor.java:64)
at jetbrains.buildServer.serverSide.stat.StatisticsMessageProcessor.processMessage(StatisticsMessageProcessor.java:6)
at jetbrains.buildServer.serverSide.impl.RunningBuildImpl.addBuildMessage(RunningBuildImpl.java:189)
at jetbrains.buildServer.serverSide.impl.ServerRunningBuildImpl.addBuildMessage(ServerRunningBuildImpl.java:29)
at jetbrains.buildServer.serverSide.impl.XmlRpcBasedServer$1.doInTransactionWithoutResult(XmlRpcBasedServer.java:9)
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:114)
at jetbrains.buildServer.serverSide.impl.XmlRpcBasedServer.log(XmlRpcBasedServer.java:43)
at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at jetbrains.buildServer.xmlrpc.ExceptionCollectorWrapper.execute(ExceptionCollectorWrapper.java:30)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at jetbrains.buildServer.controllers.XmlRpcController$1$1.execute(XmlRpcController.java)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:114)
at jetbrains.buildServer.controllers.XmlRpcController.handleRequest(XmlRpcController.java:29)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:723)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:663)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:394)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:358)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Unknown Source)
ERROR - verSide.impl.XmlRpcBasedServer - Trouble logging BuildMessage1 TestFailure jetbrains.buildServer.messages.TestProblemData@8455dd DefaultMessage

org.springframework.jdbc.UncategorizedSQLException: ConnectionCallback; uncategorized SQLException for SQL []; SQL state ; error code ; S1000 General error java.lang.NullPointerException in statement ; nested exception is java.sql.SQLException: S1000 General error java.lang.NullPointerException in statement
java.sql.SQLException: S1000 General error java.lang.NullPointerException in statement
at org.hsqldb.jdbc.Util.throwError(Unknown Source)
at org.hsqldb.jdbc.jdbcPreparedStatement.execute(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:168)
at jetbrains.buildServer.serverSide.impl.query.Query.execute(Query.java:27)
at jetbrains.buildServer.serverSide.impl.query.Query$3.run(Query.java)
at jetbrains.buildServer.serverSide.impl.DBFacade$13.doInConnection(DBFacade.java)
at jetbrains.buildServer.serverSide.impl.DBFacade$15.doInConnection(DBFacade.java)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:230)
at jetbrains.buildServer.serverSide.impl.DBFacade._runSql(DBFacade.java:219)
at jetbrains.buildServer.serverSide.impl.DBFacade.runSql(DBFacade.java:186)
at jetbrains.buildServer.serverSide.impl.DBFacade.runSql(DBFacade.java:10)
at jetbrains.buildServer.serverSide.impl.query.Query.execute(Query.java:11)
at jetbrains.buildServer.serverSide.impl.query.InsertFailedTestOutputQuery.execute(InsertFailedTestOutputQuery.java:3)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor$13.processTestProblem(AntMessagesProcessor.java:6)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor$13.processMessage(AntMessagesProcessor.java:2)
at jetbrains.buildServer.serverSide.stat.AntMessagesProcessor.doProcessMessage(AntMessagesProcessor.java:64)
at jetbrains.buildServer.serverSide.stat.StatisticsMessageProcessor.processMessage(StatisticsMessageProcessor.java:6)
at jetbrains.buildServer.serverSide.impl.RunningBuildImpl.addBuildMessage(RunningBuildImpl.java:189)
at jetbrains.buildServer.serverSide.impl.ServerRunningBuildImpl.addBuildMessage(ServerRunningBuildImpl.java:29)
at jetbrains.buildServer.serverSide.impl.XmlRpcBasedServer$1.doInTransactionWithoutResult(XmlRpcBasedServer.java:9)
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:114)
at jetbrains.buildServer.serverSide.impl.XmlRpcBasedServer.log(XmlRpcBasedServer.java:43)
at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at jetbrains.buildServer.xmlrpc.ExceptionCollectorWrapper.execute(ExceptionCollectorWrapper.java:30)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at jetbrains.buildServer.controllers.XmlRpcController$1$1.execute(XmlRpcController.java)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:114)
at jetbrains.buildServer.controllers.XmlRpcController.handleRequest(XmlRpcController.java:29)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:723)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:663)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:394)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:358)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Unknown Source)
ERROR - verSide.impl.XmlRpcBasedServer - Trouble logging BuildMessage1 BlockEnd $TEST_BLOCK$ testSortBySortFields DefaultMessage

3 comments
Comment actions Permalink

I have since update the heap to 256-512 on team city, but i am still seeing the errors. I have also noticed that the build agent can not connect at all. Here is the build agent exception.

DEBUG - rver.agent.impl.BuildAgentImpl - jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://tpteamcity:7000/RPC2 buildServer.register: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.springframework.jdbc.UncategorizedSQLException: ConnectionCallback; uncategorized SQLException for SQL []; SQL state ; error code ; out of memory; nested exception is java.sql.SQLException: out of memory
jetbrains.buildServer.xmlrpc.RemoteCallException: Call http://tpteamcity:7000/RPC2 buildServer.register: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.springframework.jdbc.UncategorizedSQLException: ConnectionCallback; uncategorized SQLException for SQL []; SQL state ; error code ; out of memory; nested exception is java.sql.SQLException: out of memory
at jetbrains.buildServer.xmlrpc.AbstractXmlRpcTarget.call(AbstractXmlRpcTarget.java:60)
at jetbrains.buildServer.agent.impl.ServerXmlRpcProxy.register(ServerXmlRpcProxy.java:46)
at jetbrains.buildServer.agent.impl.BuildAgentImpl.registerOnBuildServer(BuildAgentImpl.java:1129)
at jetbrains.buildServer.agent.impl.ServerMonitor.run(ServerMonitor.java:53)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.springframework.jdbc.UncategorizedSQLException: ConnectionCallback; uncategorized SQLException for SQL []; SQL state ; error code ; out of memory; nested exception is java.sql.SQLException: out of memory
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.TCXmlRpcClient$1.execute(TCXmlRpcClient.java:65)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at jetbrains.buildServer.xmlrpc.AbstractXmlRpcTarget.call(AbstractXmlRpcTarget.java:54)
... 3 more

0
Comment actions Permalink

Brandon, I'm afraid I've got bad news for you. You've encountered HSQLDB database corruption, and I'm not sure there is a way to recover from this problem (unless you have a backup).

You can recreate database from scratch, in this case you'll loose information about builds history and user settings. Projects and build configuration settings are stored outside of DB, so they are not affected.

To recreate database you'll have to remove content of .BuildServer/system directory excluding license.keys file, and restart build server.

You may also consider using Mysql DB for your data, please find instructions at
http://www.jetbrains.net/confluence/display/TCD/ref_installation#ref_installation-SettingupMySQL

Sorry for inconvenience,
KIR

--
Kirill Maximov
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

Ouch, that is what I was afraid of. Okay, we just a new server that we will be using for TeamCity, I'll make sure to switch to mySql.

Thanks.
-Brandon

0

Please sign in to leave a comment.