Can't natvigate a project page and ghost queue inTeamCity

Hi,

I have encounter some weired issues in TeamCity.

1.) I was not able to natvigate to TeamCity's project page. I can goto the TeamCity homepage (http://w2dconfig01:8000/overview.html) without a problem, but when I click into any of the projects (e.g. http://w2dconfig01:8000/viewType.html?buildTypeId=bt290&tab=buildTypeStatusDiv), it will just keep loading without returning anything. If I goto the config project page via "edit settings" on those projects, I can enter the config page.

2.) There're 3 TeamCity tasks that have "1 build in queue" but not processing anything. When I try to mouseover on the dropdown of that queue, it returns nothing. I have checked the status of the 3 build agents and they are all available. I suspect it may related to issue 1 as mentioned above.

I have attached the server log here. If you need more explaination on my issue please let me know. Thanks in advance.

Kind regards,
Thomas

16 comments
Comment actions Permalink

I have attached a few screenshots here.

001.JPG

There's a queue trigger by VCS check-in

002.JPG

When I mouseover to expand the queue info box, it is not returning anything...

003.JPG

The "Build Queue" tap on top is not responding to my click

004.JPG

And the task within the project as well, not responding to my click

005.JPG

However, I can goto the task admin page by clicking "edit setting"

006.JPG
The admin page load successfully

0
Comment actions Permalink

Hi,

FYI, I have recently removed a few TeamCity Projects / Tasks / VCS, and after that TeamCity webpage unable to open and I did restart the service and the server. The TeamCity page is back but with this current issue.

EDIT: This is TeamCity Enterprise Version 4.5.4 (build 9071) with subversion 1.6


Regards,
Thomas



Attachment(s):
teamcity_log_20091116.zip
0
Comment actions Permalink

Tomas,

Can you please take a thread dump of the server for the pages that you have experienced the problem and attach it?

0
Comment actions Permalink

As attached. Thanks.



Attachment(s):
TeamCityDump_20091117.log.zip
0
Comment actions Permalink

Thomas,

could you please look at your MS SQL server -
while the system is in stuck, are connections from TeamCity active or not?
Are CPUs utilized and does a disk LED light?

Thanks,
Leonid

0
Comment actions Permalink

Thomas,

Seems, TeamCity is locking on database access. Do you know is there some configuration/etc. change at the time you started to experience the issue?

TeamCity can add some extra database load while indexing tests for searching by their names, etc. If you do not need the search, you can try to disable it by specifying the following JVM options for the TeamCity server:

-Dtc.search.disable.initial.index=true
-Dtc.search.index.tests=false

If the database is functions OK for other applications, we might also need TeamCity server logs to look into the issue.

0
Comment actions Permalink

Hi,

This morning when I back to work, TeamCity is unavailable even to its homepage.

001.JPG

Before I restart TeamCity service, I have checked the DB server of TeamCity and the TeamCity host, and I see no heavy loading on CPU, and no sign of disconnection too.
Left hand side is the DB server and right hand side is the TeamCity host.

002.JPG

Then I restart the TeamCity service on the host, then TeamCity homepage works again, with the previous problem, where the queue is not showing up and sub-project pages are not loading properly.

003.JPG

I have again attached the TeamCity log here. If there's additional info needed please let me know. Thanks.

Kind regards,
Thomas



Attachment(s):
TeamCity.Log.2009-11-18.zip
0
Comment actions Permalink

Hi,

I believe there's no changes on TeamCity configuration as it will usually be me who make changes on TeamCity, and I have setup another host for TeamCity EAP and that is on the same DB server, where the EAP host seems to have no problem with database connection.

Please refer to my previous post for attached TeamCity log. Thank you very much for your help.

Regards,
Thomas

0
Comment actions Permalink

The following are some extracts of the error:

[2009-11-18 10:01:29,813]  ERROR -   jetbrains.buildServer.SERVER - Error occurred while starting build

org.springframework.jdbc.UncategorizedSQLException: Error preparing statement; uncategorized SQLException for SQL [update server set build_id = ?]; SQL state [08S01]; error code [0]; Network error IOException: Connection refused: connect; nested exception is java.sql.SQLException: Network error IOException: Connection refused: connect
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.getPreparedStatement(GenericQuery.java:14)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.doExecuteQuery(GenericQuery.java:31)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.access$100(GenericQuery.java:123)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery$1.run(GenericQuery.java:1)
     at jetbrains.buildServer.serverSide.impl.DBFacade$3.doInConnection(DBFacade.java:1)
     at jetbrains.buildServer.serverSide.impl.DBFacade$4.doInConnection(DBFacade.java:2)
     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:341)
     at jetbrains.buildServer.serverSide.impl.DBFacade._runSql(DBFacade.java:27)
     at jetbrains.buildServer.serverSide.impl.DBFacade.runSql(DBFacade.java:203)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.execute(GenericQuery.java:45)
     at jetbrains.buildServer.serverSide.impl.BuildIdGenerator.updateId(BuildIdGenerator.java:10)
     at jetbrains.buildServer.serverSide.impl.BuildIdGenerator.getNextId(BuildIdGenerator.java:5)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl.startBuild(BuildServerImpl.java:494)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl.access$700(BuildServerImpl.java:230)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl$5.run(BuildServerImpl.java:13)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.executeWithInactiveCleanup(ServerCleanupManagerImpl.java:109)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$FastClassByCGLIB$$ba2c8525.invoke(<generated>)
     at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
     at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
     at jetbrains.buildServer.serverSide.impl.auth.TeamCityMethodSecurityInterceptor.invoke(TeamCityMethodSecurityInterceptor.java:8)
     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
     at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:582)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$EnhancerByCGLIB$$dc8f8365.executeWithInactiveCleanup(<generated>)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl.flushQueue(BuildServerImpl.java:222)
     at jetbrains.buildServer.serverSide.impl.BuildServerRunner$2.doSomething(BuildServerRunner.java:2)
     at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:0)
     at java.lang.Thread.run(Thread.java:595)
Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:385)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
     at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:182)
     at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
     at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
     at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
     at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
     at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
     at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
     at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:210)
     at $Proxy8.prepareStatement(Unknown Source)
     at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:585)
     at org.springframework.jdbc.core.JdbcTemplate$CloseSuppressingInvocationHandler.invoke(JdbcTemplate.java:1279)
     at $Proxy8.prepareStatement(Unknown Source)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.getPreparedStatement(GenericQuery.java:91)
     ... 27 more
Caused by: java.net.ConnectException: Connection refused: connect
     at java.net.PlainSocketImpl.socketConnect(Native Method)
     at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
     at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
     at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
     at java.net.Socket.connect(Socket.java:519)
     at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:585)
     at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:304)
     at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:255)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:310)
     ... 43 more
[2009-11-18 10:01:36,235]  ERROR -   jetbrains.buildServer.SERVER - Error occurred while starting build

org.springframework.jdbc.UncategorizedSQLException: Error preparing statement; uncategorized SQLException for SQL [update server set build_id = ?]; SQL state [08S01]; error code [0]; Network error IOException: Connection refused: connect; nested exception is java.sql.SQLException: Network error IOException: Connection refused: connect
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.getPreparedStatement(GenericQuery.java:14)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.doExecuteQuery(GenericQuery.java:31)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.access$100(GenericQuery.java:123)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery$1.run(GenericQuery.java:1)
     at jetbrains.buildServer.serverSide.impl.DBFacade$3.doInConnection(DBFacade.java:1)
     at jetbrains.buildServer.serverSide.impl.DBFacade$4.doInConnection(DBFacade.java:2)
     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:341)
     at jetbrains.buildServer.serverSide.impl.DBFacade._runSql(DBFacade.java:27)
     at jetbrains.buildServer.serverSide.impl.DBFacade.runSql(DBFacade.java:203)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.execute(GenericQuery.java:45)
     at jetbrains.buildServer.serverSide.impl.BuildIdGenerator.updateId(BuildIdGenerator.java:10)
     at jetbrains.buildServer.serverSide.impl.BuildIdGenerator.getNextId(BuildIdGenerator.java:5)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl.startBuild(BuildServerImpl.java:494)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl.access$700(BuildServerImpl.java:230)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl$5.run(BuildServerImpl.java:13)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.executeWithInactiveCleanup(ServerCleanupManagerImpl.java:109)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$FastClassByCGLIB$$ba2c8525.invoke(<generated>)
     at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
     at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)
     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
     at jetbrains.buildServer.serverSide.impl.auth.TeamCityMethodSecurityInterceptor.invoke(TeamCityMethodSecurityInterceptor.java:8)
     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
     at org.springframework.aop.framework.Cglib2AopProxy$FixedChainStaticTargetInterceptor.intercept(Cglib2AopProxy.java:582)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$EnhancerByCGLIB$$dc8f8365.executeWithInactiveCleanup(<generated>)
     at jetbrains.buildServer.serverSide.impl.BuildServerImpl.flushQueue(BuildServerImpl.java:222)
     at jetbrains.buildServer.serverSide.impl.BuildServerRunner$2.doSomething(BuildServerRunner.java:2)
     at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:0)
     at java.lang.Thread.run(Thread.java:595)
Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:385)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC3.<init>(ConnectionJDBC3.java:50)
     at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:182)
     at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
     at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
     at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:840)
     at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
     at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
     at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
     at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:210)
     at $Proxy8.prepareStatement(Unknown Source)
     at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:585)
     at org.springframework.jdbc.core.JdbcTemplate$CloseSuppressingInvocationHandler.invoke(JdbcTemplate.java:1279)
     at $Proxy8.prepareStatement(Unknown Source)
     at jetbrains.buildServer.serverSide.impl.query.GenericQuery.getPreparedStatement(GenericQuery.java:91)
     ... 27 more
Caused by: java.net.ConnectException: Connection refused: connect
     at java.net.PlainSocketImpl.socketConnect(Native Method)
     at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
     at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
     at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
     at java.net.Socket.connect(Socket.java:519)
     at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:585)
     at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:304)
     at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:255)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.<init>(ConnectionJDBC2.java:310)
     ... 43 more


And seems I found the source of the error:

[2009-11-13 03:01:25,043]   INFO -  jetbrains.buildServer.CLEANUP - Builds to clean for cleanup level (Clean everything):      720 builds: [9136, 9138, 9139, 9152, 9154, 9155, 9160, 9162, 9163, 9166, 9178, 9189, 9191, 9194, 9195, 9198, 9517, 9518, 9526, 9528, 9570, 9572, 9583, 9631, 9634, 9639, 9642, 9647, 9650, 9665, 9666, 9667, 9668, 9669, 9670, 9671, 9672, 9676, 9679, 9682, 9686, 9690, 9744, 9745, 9749, 9750, 9756, 9758, 9763, 9772, 9773, 9774, 9775, 9776, 9777, 9782, 9786, 9788, 9792, 9793, 9804, 9806, 9810, 9848, 9849, 9853, 9854, 9859, 9865, 9868, 9873, 9876, 9880, 9881, 9886, 9887, 9891, 9902, 9903, 9907, 9909, 9912, 9916, 9917, 9926, 9927, 9931, 9933, 9937, 9975, 9979, 10008, 10012, 10029, 10032, 10039, 10042, 10078, 10081, 10122, 10150, 10151, 10158, 10159, 10164, 10272, 10277, 10343, 10348, 10352, 10356, 10360, 10364, 10382, 10391, 10406, 10407, 10413, 10428, 10430, 10431, 10432, 10445, 10454, 10455, 10456, 10459, 10462, 10463, 10467, 10468, 10475, 10476, 10477, 10478, 10492, 10493, 10498, 10499, 10500, 10504, 10505, 10507, 10510, 10511, 10512, 10513, 10522, 10535, 10537, 10546, 10549, 10552, 10553, 10563, 10564, 10565, 10568, 10569, 10570, 10571, 10572, 10577, 10585, 10593, 10594, 10595, 10596, 10597, 10600, 10601, 10602, 10603, 10604, 10610, 10611, 10612, 10615, 10621, 10622, 10623, 10626, 10630, 10631, 10634, 10637, 10638, 10693, 10695, 10697, 10703, 10704, 10710, 10711, 10734, 10740, 10741, 10745, 10747, 10750, 10751, 10754, 10761, 10762, 10769, 10770, 10772, 10776, 10787, 10788, 10789, 10799, 10807, 10808, 10809, 10812, 10813, 10814, 10815, 10817, 10836, 10842, 10844, 10856, 10861, 10862, 10871, 10878, 10879, 10881, 10889, 10890, 10891, 10892, 10893, 10895, 10897, 10898, 10907, 10935, 10936, 10937, 10939, 10941, 10942, 10943, 10948, 10959, 10960, 10961, 10962, 10970, 10971, 10973, 10974, 10975, 10976, 10978, 10983, 10984, 10986, 10992, 10993, 10995, 11007, 11031, 11032, 11034, 11039, 11040, 11042, 11046, 11047, 11049, 11050, 11052, 11054, 11081, 11085, 11097, 11098, 11099, 11100, 11102, 11106, 11115, 11117, 11119, 11120, 11122, 11125, 11129, 11130, 11132, 11133, 11134, 11135, 11146, 11147, 11148, 11151, 11152, 11153, 11154, 11155, 11156, 11159, 11166, 12158, 12159, 12545, 12776, 12777, 12784, 12786, 12789, 12790, 12791, 12792, 12793, 12794, 12796, 12802, 12803, 12804, 12812, 12814, 12816, 12818, 12819, 12820, 12821, 12825, 12831, 12832, 12833, 12834, 12835, 12843, 12844, 12846, 12847, 12848, 12849, 12850, 12852, 12853, 12858, 12859, 12860, 12862, 12863, 12864, 12940, 12946, 12947, 12948, 12954, 12955, 12956, 12961, 12962, 12966, 12967, 12969, 12970, 12972, 12973, 12974, 12975, 12983, 12985, 12986, 12988, 12995, 12999, 13001, 13002, 13004, 13005, 13010, 13011, 13018, 13020, 13068, 13069, 13076, 13081, 13082, 13089, 13099, 13100, 13102, 13110, 13112, 13113, 13115, 13116, 13118, 13119, 13125, 13126, 13127, 13134, 13136, 13160, 13161, 13163, 13184, 13185, 13186, 13187, 13188, 13189, 13190, 13192, 13193, 13200, 13205, 13212, 13213, 13214, 13218, 13220, 13221, 13223, 13225, 13226, 13272, 13278, 13279, 13281, 13287, 13288, 13294, 13295, 13297, 13302, 13307, 13314, 13315, 13316, 13357, 13359, 13364, 13369, 13370, 13371, 13372, 13374, 13376, 13377, 13390, 13391, 13397, 13398, 13401, 13403, 13404, 13405, 13406, 13435, 13442, 13451, 13457, 13480, 13485, 13488, 13493, 13494, 13495, 13498, 13505, 13506, 13509, 13510, 13511, 13512, 13513, 13514, 13556, 13570, 13585, 13586, 13587, 13591, 13592, 13593, 13597, 13598, 13599, 13623, 13624, 13625, 13628, 13629, 13630, 13631, 13632, 13633, 13634, 13640, 13641, 13646, 13648, 13651, 13653, 13655, 13659, 13661, 13662, 13663, 13664, 13665, 13668, 13676, 13677, 13678, 13680, 13681, 13682, 13684, 13685, 13687, 13688, 13692, 13693, 13708, 13709, 13710, 13712, 13713, 13715, 13746, 13748, 13808, 13809, 13810, 13811, 13818, 13844, 13848, 13849, 13868, 13879, 13880, 13900, 13901, 13902, 13903, 13904, 13905, 13906, 13921, 13936, 13940, 13941, 13955, 13956, 13958, 13979, 13984, 13992, 13993, 13994, 14041, 14046, 14117, 14123, 14128, 14133, 14142, 14154, 14155, 14156, 14174, 14179, 14220, 14259, 14284, 14289, 14318, 14320, 14321, 14337, 14339, 14340, 14345, 14347, 14349, 14350, 14352, 14359, 14364, 14424, 14426, 14427, 14432, 14434, 14466, 14468, 14508, 14510, 14511, 14520, 14521, 14523, 14524, 14526, 14535, 14537, 14538, 14540, 14542, 14544, 14546, 14551, 14556, 14588, 14590, 14591, 14593, 14601, 14612, 14615, 14621, 14623, 14637, 14639, 14655, 14657, 14665, 14697, 14737, 14738, 14767, 14768, 14770, 14771, 14773, 14883, 14888, 14917, 14918, 14925, 14930, 14931, 14933, 14934, 14936, 14950, 14953, 14982, 14984, 15019, 15028, 15032, 15035, 15037, 15087, 15089, 15091, 15101, 15103, 15135, 15137, 15138, 15139, 15144, 15176, 15177, 15178, 15186, 15191, 15197, 15198, 15206, 15207, 15208, 15209, 15210, 15211, 15212, 15231, 15239, 15250, 15257, 15258, 15285, 15363, 15381, 15420, 15429, 15435, 15440, 15476, 15478, 15489, 15498, 15508, 15865, 17130, 17137, 17140, 17149, 17370, 17377, 17382, 17435, 17673, 17679, 17824, 17827, 18104, 19472, 19538, 19539, 19540, 19541, 19542]
[2009-11-13 03:01:25,058]   INFO -  jetbrains.buildServer.CLEANUP - Builds to clean for cleanup level (Clean artifacts):      1611 builds: [1135, 1145, 1162, 1164, 1805, 1806, 1807, 1808, 1809, 1810, 1833, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1846, 1847, 1848, 1849, 1850, 1851, 1868, 1869, 2585, 2947, 7567, 7568, 9173, 9771, 9870, 9871, 9872, 9874, 9878, 9879, 9882, 9883, 9884, 9885, 9888, 9890, 9893, 9894, 9895, 9896, 9898, 9899, 9900, 9901, 9904, 9905, 9906, 9908, 9911, 9914, 9915, 9918, 9919, 9920, 9921, 9922, 9923, 9924, 9925, 9928, 9929, 9930, 9932, 9935, 9939, 9941, 9942, 9943, 9944, 9945, 9946, 9947, 9948, 9951, 9952, 9953, 9954, 9958, 9960, 9961, 9962, 9963, 9965, 9966, 9967, 9969, 9970, 9971, 9972, 9974, 9977, 9978, 9981, 9982, 9983, 9984, 9985, 9986, 9987, 9988, 9989, 9991, 9992, 9993, 9994, 9996, 9997, 9998, 9999, 10001, 10002, 10003, 10004, 10005, 10006, 10009, 10011, 10015, 10016, 10017, 10018, 10019, 10021, 10022, 10023, 10024, 10025, 10026, 10027, 10030, 10035, 10036, 10037, 10038, 10041, 10045, 10047, 10049, 10051, 10052, 10054, 10055, 10057, 10058, 10059, 10060, 10061, 10062, 10063, 10064, 10065, 10066, 10067, 10068, 10069, 10070, 10071, 10072, 10073, 10074, 10075, 10077, 10080, 10083, 10084, 10086, 10087, 10088, 10090, 10093, 10094, 10095, 10096, 10097, 10098, 10099, 10100, 10101, 10102, 10103, 10104, 10105, 10106, 10107, 10108, 10109, 10110, 10111, 10112, 10113, 10114, 10115, 10116, 10117, 10119, 10120, 10121, 10124, 10125, 10126, 10127, 10129, 10130, 10131, 10132, 10134, 10135, 10136, 10137, 10138, 10139, 10140, 10141, 10142, 10146, 10147, 10149, 10152, 10153, 10154, 10156, 10160, 10162, 10165, 10166, 10167, 10168, 10169, 10170, 10171, 10172, 10174, 10175, 10176, 10177, 10178, 10179, 10180, 10182, 10183, 10186, 10187, 10188, 10189, 10191, 10192, 10193, 10194, 10195, 10196, 10197, 10198, 10199, 10200, 10201, 10202, 10203, 10204, 10205, 10206, 10207, 10208, 10209, 10210, 10211, 10212, 10213, 10215, 10216, 10217, 10218, 10219, 10220, 10221, 10222, 10224, 10225, 10226, 10227, 10228, 10229, 10230, 10231, 10232, 10234, 10235, 10236, 10237, 10238, 10239, 10241, 10242, 10243, 10244, 10245, 10246, 10247, 10248, 10249, 10251, 10252, 10254, 10255, 10256, 10257, 10258, 10259, 10260, 10261, 10262, 10264, 10266, 10267, 10268, 10269, 10270, 10273, 10274, 10278, 10280, 10281, 10282, 10283, 10284, 10285, 10287, 10288, 10289, 10290, 10292, 10293, 10294, 10295, 10296, 10298, 10299, 10301, 10303, 10304, 10305, 10306, 10307, 10308, 10309, 10310, 10311, 10312, 10313, 10314, 10315, 10316, 10317, 10318, 10320, 10321, 10322, 10323, 10324, 10325, 10328, 10329, 10330, 10331, 10333, 10334, 10336, 10337, 10338, 10339, 10340, 10341, 10344, 10345, 10349, 10350, 10351, 10353, 10357, 10358, 10359, 10361, 10365, 10367, 10368, 10369, 10370, 10371, 10372, 10373, 10374, 10375, 10376, 10377, 10378, 10379, 10380, 10381, 10383, 10384, 10385, 10386, 10388, 10389, 10390, 10392, 10393, 10394, 10395, 10396, 10397, 10398, 10399, 10400, 10401, 10402, 10403, 10408, 10409, 10411, 10414, 10415, 10416, 10417, 10418, 10419, 10420, 10421, 10424, 10425, 10426, 10427, 10429, 10433, 10434, 10435, 10436, 10437, 10438, 10439, 10440, 10441, 10442, 10443, 10444, 10447, 10449, 10450, 10451, 10452, 10453, 10457, 10458, 10460, 10461, 10464, 10465, 10466, 10469, 10470, 10471, 10472, 10473, 10474, 10479, 10480, 10481, 10482, 10483, 10484, 10485, 10486, 10487, 10488, 10489, 10490, 10491, 10494, 10495, 10496, 10497, 10501, 10502, 10503, 10506, 10508, 10509, 10514, 10515, 10516, 10517, 10518, 10519, 10520, 10521, 10536, 10538, 10539, 10540, 10541, 10542, 10543, 10544, 10545, 10547, 10548, 10550, 10551, 10554, 10555, 10556, 10557, 10559, 10560, 10566, 10567, 10574, 10575, 10578, 10579, 10580, 10581, 10582, 10583, 10586, 10587, 10588, 10589, 10590, 10591, 10599, 10605, 10606, 10607, 10608, 10613, 10614, 10616, 10617, 10618, 10619, 10624, 10625, 10627, 10628, 10629, 10632, 10633, 10635, 10636, 10639, 10640, 10641, 10642, 10643, 10644, 10645, 10646, 10647, 10648, 10649, 10651, 10652, 10653, 10654, 10655, 10656, 10657, 10660, 10661, 10662, 10663, 10664, 10665, 10666, 10668, 10669, 10670, 10671, 10672, 10673, 10675, 10677, 10678, 10679, 10680, 10681, 10682, 10683, 10684, 10685, 10686, 10687, 10688, 10689, 10690, 10691, 10694, 10696, 10699, 10700, 10701, 10702, 10705, 10706, 10707, 10708, 10709, 10712, 10713, 10714, 10715, 10716, 10717, 10718, 10719, 10720, 10721, 10722, 10723, 10725, 10727, 10728, 10729, 10730, 10731, 10732, 10733, 10735, 10736, 10737, 10738, 10739, 10742, 10743, 10744, 10746, 10748, 10749, 10752, 10753, 10755, 10756, 10757, 10758, 10759, 10760, 10763, 10764, 10765, 10766, 10767, 10768, 10773, 10774, 10775, 10777, 10778, 10779, 10780, 10781, 10782, 10783, 10784, 10785, 10786, 10791, 10792, 10793, 10794, 10795, 10796, 10797, 10800, 10801, 10802, 10803, 10804, 10805, 10806, 10810, 10816, 10818, 10819, 10820, 10821, 10822, 10823, 10824, 10825, 10827, 10828, 10830, 10831, 10832, 10833, 10834, 10835, 10837, 10838, 10839, 10840, 10841, 10843, 10845, 10846, 10847, 10848, 10849, 10851, 10853, 10854, 10855, 10857, 10858, 10859, 10860, 10864, 10865, 10866, 10867, 10868, 10869, 10870, 10872, 10874, 10875, 10876, 10877, 10880, 10882, 10883, 10884, 10885, 10886, 10888, 10894, 10896, 10899, 10900, 10901, 10902, 10903, 10904, 10905, 10906, 10909, 10910, 10912, 10913, 10914, 10915, 10916, 10917, 10918, 10920, 10921, 10922, 10923, 10924, 10926, 10927, 10928, 10929, 10930, 10931, 10932, 10933, 10934, 10938, 10940, 10944, 10945, 10946, 10947, 10950, 10952, 10953, 10956, 10957, 10964, 10965, 10966, 10967, 10968, 10969, 10972, 10977, 10979, 10980, 10981, 10982, 10985, 10987, 10988, 10989, 10990, 10991, 10994, 10997, 10998, 11001, 11008, 11009, 11010, 11012, 11013, 11015, 11016, 11017, 11018, 11019, 11020, 11021, 11022, 11023, 11024, 11027, 11028, 11029, 11030, 11033, 11035, 11036, 11037, 11041, 11043, 11044, 11045, 11048, 11051, 11053, 11055, 11056, 11057, 11058, 11059, 11060, 11061, 11062, 11063, 11064, 11065, 11066, 11067, 11068, 11069, 11070, 11071, 11072, 11073, 11074, 11075, 11076, 11077, 11078, 11080, 11082, 11083, 11084, 11086, 11087, 11088, 11089, 11090, 11091, 11092, 11093, 11094, 11095, 11096, 11101, 11103, 11104, 11105, 11107, 11108, 11110, 11111, 11112, 11113, 11114, 11116, 11118, 11121, 11124, 11126, 11127, 11128, 11131, 11136, 11137, 11138, 11139, 11145, 11150, 11158, 11160, 11161, 11162, 11164, 11165, 11184, 11193, 11955, 11956, 11957, 12544, 12775, 12795, 12797, 12798, 12799, 12800, 12801, 12806, 12807, 12808, 12811, 12813, 12815, 12817, 12822, 12823, 12824, 12826, 12827, 12828, 12829, 12836, 12838, 12839, 12840, 12842, 12845, 12851, 12854, 12855, 12856, 12857, 12878, 12887, 12888, 12889, 12890, 12891, 12898, 12911, 12912, 12913, 12914, 12916, 12917, 12941, 12942, 12943, 12944, 12949, 12950, 12951, 12952, 12957, 12958, 12959, 12960, 12963, 12964, 12965, 12968, 12971, 12994, 12996, 12997, 12998, 13000, 13003, 13006, 13007, 13008, 13009, 13019, 13063, 13064, 13067, 13077, 13078, 13079, 13080, 13101, 13111, 13114, 13117, 13120, 13121, 13122, 13123, 13135, 13155, 13162, 13191, 13201, 13202, 13203, 13204, 13206, 13207, 13208, 13209, 13210, 13211, 13216, 13217, 13219, 13228, 13232, 13233, 13234, 13238, 13239, 13240, 13241, 13244, 13260, 13261, 13262, 13263, 13282, 13283, 13284, 13285, 13286, 13298, 13299, 13300, 13301, 13303, 13304, 13305, 13306, 13308, 13309, 13310, 13311, 13312, 13313, 13317, 13318, 13319, 13320, 13321, 13334, 13335, 13336, 13343, 13344, 13345, 13346, 13347, 13348, 13349, 13356, 13358, 13360, 13361, 13362, 13363, 13365, 13366, 13367, 13368, 13373, 13375, 13378, 13379, 13380, 13381, 13382, 13383, 13384, 13388, 13389, 13399, 13402, 13420, 13421, 13441, 13444, 13445, 13446, 13447, 13448, 13449, 13450, 13452, 13453, 13454, 13455, 13456, 13458, 13459, 13460, 13461, 13468, 13469, 13470, 13471, 13472, 13473, 13474, 13475, 13476, 13477, 13478, 13479, 13481, 13482, 13483, 13484, 13486, 13487, 13489, 13490, 13491, 13492, 13497, 13499, 13508, 13515, 13516, 13517, 13524, 13557, 13558, 13559, 13560, 13561, 13562, 13563, 13564, 13565, 13566, 13567, 13568, 13569, 13576, 13580, 13582, 13583, 13584, 13588, 13589, 13590, 13594, 13595, 13596, 13607, 13609, 13635, 13636, 13637, 13638, 13639, 13642, 13643, 13644, 13645, 13647, 13652, 13654, 13656, 13657, 13658, 13660, 13667, 13671, 13672, 13673, 13674, 13675, 13679, 13683, 13686, 13689, 13690, 13691, 13694, 13695, 13696, 13697, 13711, 13714, 13716, 13717, 13718, 13724, 13725, 13726, 13727, 13729, 13730, 13731, 13733, 13744, 13747, 13762, 13772, 13773, 13812, 13813, 13814, 13815, 13816, 13846, 13850, 13851, 13854, 13855, 13856, 13857, 13870, 13871, 13882, 13883, 13884, 13899, 13907, 13908, 13909, 13910, 13911, 13916, 13917, 13918, 13919, 13920, 13922, 13923, 13924, 13925, 13939, 13945, 13946, 13947, 13957, 13959, 13960, 13961, 13962, 13963, 13964, 13971, 13978, 13980, 13981, 13982, 13983, 13990, 14001, 14006, 14040, 14042, 14043, 14044, 14045, 14078, 14080, 14081, 14085, 14096, 14100, 14101, 14113, 14114, 14118, 14119, 14120, 14121, 14122, 14124, 14125, 14126, 14127, 14129, 14130, 14131, 14132, 14134, 14135, 14136, 14137, 14138, 14139, 14140, 14141, 14143, 14144, 14145, 14146, 14147, 14148, 14149, 14157, 14158, 14159, 14160, 14161, 14162, 14163, 14164, 14165, 14166, 14175, 14176, 14177, 14178, 14180, 14183, 14184, 14185, 14187, 14188, 14189, 14193, 14194, 14195, 14202, 14204, 14205, 14207, 14208, 14211, 14213, 14214, 14215, 14216, 14217, 14218, 14219, 14221, 14222, 14223, 14224, 14225, 14226, 14227, 14228, 14240, 14242, 14243, 14244, 14245, 14252, 14253, 14254, 14256, 14267, 14268, 14269, 14270, 14271, 14272, 14273, 14280, 14281, 14282, 14285, 14286, 14287, 14288, 14290, 14291, 14292, 14293, 14294, 14295, 14296, 14298, 14304, 14305, 14306, 14307, 14308, 14309, 14310, 14311, 14312, 14319, 14322, 14323, 14324, 14325, 14338, 14344, 14348, 14351, 14360, 14361, 14362, 14363, 14370, 14371, 14375, 14376, 14398, 14408, 14409, 14410, 14411, 14412, 14413, 14414, 14420, 14421, 14422, 14423, 14425, 14428, 14429, 14430, 14431, 14433, 14467, 14500, 14501, 14509, 14512, 14513, 14514, 14515, 14517, 14518, 14519, 14522, 14525, 14527, 14528, 14529, 14532, 14533, 14534, 14536, 14539, 14543, 14545, 14547, 14548, 14549, 14550, 14552, 14553, 14554, 14555, 14557, 14558, 14559, 14560, 14561, 14562, 14563, 14564, 14565, 14566, 14567, 14568, 14569, 14573, 14574, 14575, 14589, 14592, 14602, 14603, 14604, 14605, 14613, 14614, 14622, 14638, 14656, 14658, 14659, 14660, 14661, 14662, 14663, 14664, 14666, 14667, 14668, 14669, 14692, 14693, 14694, 14698, 14699, 14700, 14701, 14705, 14739, 14745, 14747, 14765, 14769, 14772, 14774, 14775, 14776, 14777, 14783, 14784, 14787, 14819, 14825, 14831, 14838, 14849, 14855, 14867, 14881, 14882, 14884, 14885, 14886, 14887, 14889, 14890, 14891, 14892, 14893, 14900, 14913, 14919, 14921, 14922, 14923, 14924, 14926, 14927, 14928, 14929, 14932, 14935, 14937, 14952, 14983, 15020, 15021, 15022, 15023, 15024, 15031, 15036, 15074, 15083, 15084, 15085, 15088, 15092, 15093, 15094, 15095, 15096, 15098, 15099, 15100, 15102, 15136, 15145, 15146, 15147, 15148, 15149, 15155, 15156, 15157, 15158, 15179, 15180, 15181, 15182, 15183, 15188, 15189, 15190, 15193, 15194, 15195, 15196, 15214, 15215, 15216, 15217, 15252, 15253, 15254, 15255, 15287, 15288, 15289, 15375, 15385, 19473, 19543, 19544]
[2009-11-13 03:01:25,058]   INFO -  jetbrains.buildServer.CLEANUP - Performing cleanup EVERYTHING for 720 builds
[2009-11-13 03:01:25,058]   INFO -  jetbrains.buildServer.CLEANUP - Cleaning extensions data...
[2009-11-13 03:01:33,387]   INFO -  jetbrains.buildServer.CLEANUP - Deleting build logs...
[2009-11-13 03:01:46,466]   INFO -  jetbrains.buildServer.CLEANUP - Cleaning build history data from the database...
[2009-11-13 03:02:33,922]  ERROR -  jetbrains.buildServer.CLEANUP - Unexpected error occured during cleanup

org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Invalid state, the Connection object is closed.
     at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:279)
     at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:800)
     at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:777)
     at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:154)
     at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:132)
     at jetbrains.buildServer.serverSide.impl.DBFacade.compact(DBFacade.java:43)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:96)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:5)
     at java.util.TimerThread.mainLoop(Timer.java:512)
     at java.util.TimerThread.run(Timer.java:462)
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1634)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.rollback(ConnectionJDBC2.java:2027)
     at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
     at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
     at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
     at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:276)
     ... 9 more
[2009-11-13 03:02:34,391]   INFO -   jetbrains.buildServer.SERVER - Optimizing search index...
[2009-11-13 03:02:34,813]   INFO -   jetbrains.buildServer.SERVER - Finished index optimization
[2009-11-13 07:00:04,693]   INFO -   jetbrains.buildServer.SERVER - Loading history of VCS changes...
[2009-11-13 07:00:05,053]   INFO -   jetbrains.buildServer.SERVER - Done loading VCS history: 3276 changes loaded


It looks like TeamCity clean-up results in an invalid state which affect the database connection of TeamCity. I am not 100% sure but hope that can help the investigation. Thanks.

Regards,
Thomas

0
Comment actions Permalink

Thomas, one more question: how many connections are configured in your "database.properties" file?
(it's the "maxConnections" property).

If the number of connections less than 20 please increase it up to 20.

0
Comment actions Permalink

Thomas,

> and I have setup another host for TeamCity EAP and that is on the same DB server,

am I right that you have meant TeamCity 5.0 EAP, with a new empty database?

Thanks.

0
Comment actions Permalink

Thomas,

could you please also add the two following properties into your 'database.properties' file:

  connectionProperties.validationQuery=select 1
  connectionProperties.testOnBorrow=true

and remove the "poolPreparedStatements" property if exists,
and then try again.

Thank you!

Leonid

0
Comment actions Permalink

Hi,

Just want to report that the problem was temporary solved by restarting the database instance where TeamCity host.
The reason was the TeamCity EAP server which host on another box also experience problems and I had request a restart on our database instance.

Now TeamCity server is back to a usable state, but I suspect the root cause is from the SQL Unexpected Error when TC performing the CleanUp / database maintenance.
It was happening again last night. I have capture the logs and going to restart TeamCity service. Is there a known issue of TeamCity CleanUp for version 4.5.4?

[2009-11-20 03:01:54,834]   WARN - c.support.SQLErrorCodesFactory - Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException: Invalid state, the Connection object is closed.
     at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:290)
     at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:314)
     at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:216)
     at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:147)
     at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:110)
     at org.springframework.jdbc.support.JdbcAccessor.getExceptionTranslator(JdbcAccessor.java:99)
     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:348)
     at jetbrains.buildServer.serverSide.impl.DBFacade._runSql(DBFacade.java:27)
     at jetbrains.buildServer.serverSide.impl.DBFacade.access$000(DBFacade.java:196)
     at jetbrains.buildServer.serverSide.impl.DBFacade$1.doInTransaction(DBFacade.java:2)
     at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
     at jetbrains.buildServer.serverSide.impl.DBFacade.compact(DBFacade.java:43)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:96)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:5)
     at java.util.TimerThread.mainLoop(Timer.java:512)
     at java.util.TimerThread.run(Timer.java:462)
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1634)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.getMetaData(ConnectionJDBC2.java:2213)
     at org.apache.commons.dbcp.DelegatingConnection.getMetaData(DelegatingConnection.java:310)
     at org.apache.commons.dbcp.DelegatingConnection.getMetaData(DelegatingConnection.java:310)
     at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.getMetaData(PoolingDataSource.java:234)
     at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:585)
     at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:225)
     at $Proxy8.getMetaData(Unknown Source)
     at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:279)
     ... 15 more
[2009-11-20 03:01:54,850]  ERROR - on.support.TransactionTemplate - Application exception overridden by rollback exception
org.springframework.jdbc.UncategorizedSQLException: ConnectionCallback; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; Borrow prepareStatement from pool failed; nested exception is org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from pool failed
     at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.translate(SQLStateSQLExceptionTranslator.java:124)
     at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.translate(SQLErrorCodeSQLExceptionTranslator.java:322)
     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:348)
     at jetbrains.buildServer.serverSide.impl.DBFacade._runSql(DBFacade.java:27)
     at jetbrains.buildServer.serverSide.impl.DBFacade.access$000(DBFacade.java:196)
     at jetbrains.buildServer.serverSide.impl.DBFacade$1.doInTransaction(DBFacade.java:2)
     at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:128)
     at jetbrains.buildServer.serverSide.impl.DBFacade.compact(DBFacade.java:43)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:96)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:5)
     at java.util.TimerThread.mainLoop(Timer.java:512)
     at java.util.TimerThread.run(Timer.java:462)
Caused by: org.apache.commons.dbcp.SQLNestedException: Borrow prepareStatement from pool failed
     at org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:98)
     at org.apache.commons.dbcp.DelegatingConnection.prepareStatement(DelegatingConnection.java:248)
     at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.prepareStatement(PoolingDataSource.java:302)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:585)
     at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:225)
     at $Proxy8.prepareStatement(Unknown Source)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:585)
     at org.springframework.jdbc.core.JdbcTemplate$CloseSuppressingInvocationHandler.invoke(JdbcTemplate.java:1279)
     at $Proxy8.prepareStatement(Unknown Source)
     at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.unpinBuilds(HistoryEntryCleaner.java:54)
     at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:258)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$3.performCleanup(ServerCleanupManagerImpl.java:2)
     at jetbrains.buildServer.serverSide.impl.DBFacade$1$1.doInConnection(DBFacade.java:2)
     at jetbrains.buildServer.serverSide.impl.DBFacade$4.doInConnection(DBFacade.java:2)
     at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:341)
     ... 9 more
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1634)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.prepareStatement(ConnectionJDBC2.java:2328)
     at org.apache.commons.dbcp.PoolingConnection.makeObject(PoolingConnection.java:193)
     at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:797)
     at org.apache.commons.dbcp.PoolingConnection.prepareStatement(PoolingConnection.java:92)
     ... 29 more
[2009-11-20 03:01:54,850]  ERROR -  jetbrains.buildServer.CLEANUP - Unexpected error occured during cleanup

org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Invalid state, the Connection object is closed.
     at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:279)
     at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:800)
     at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:777)
     at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:154)
     at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:132)
     at jetbrains.buildServer.serverSide.impl.DBFacade.compact(DBFacade.java:43)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:96)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:5)
     at java.util.TimerThread.mainLoop(Timer.java:512)
     at java.util.TimerThread.run(Timer.java:462)
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1634)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.rollback(ConnectionJDBC2.java:2027)
     at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
     at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
     at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
     at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:276)
     ... 9 more


[2009-11-20 03:00:07,549]   INFO -  jetbrains.buildServer.CLEANUP - Builds to clean for cleanup level (Clean everything):      3000 builds: [1155, 1156, 1157, 1165, 1166, 1249, 1253, 2585, 2630, 2876, 2947, 2951, 7823, 7824, 7825, 7828, 8080, 8086, 8087, 8088, 8089, 8090, 8097, 8107, 8176, 8266, 8267, 8268, 8276, 8332, 8334, 8335, 8506, 8507, 8508, 8618, 8643, 8644, 8645, 8646, 8647, 8648, 8649, 8653, 8654, 8655, 8656, 8657, 8658, 8659, 8660, 8661, 8662, 8663, 8711, 8712, 8713, 8714, 8735, 8736, 8741, 8748, 8749, 8762, 8771, 8773, 8774, 8775, 8776, 8777, 8780, 8781, 8782, 8783, 8810, 8811, 8812, 8813, 8814, 8821, 8822, 8823, 8824, 8825, 8826, 8827, 8828, 8844, 8846, 8847, 8848, 8876, 8877, 8878, 8879, 8880, 8881, 8882, 8883, 8885, 8886, 8887, 8888, 8889, 8890, 8939, 8943, 8944, 8945, 8946, 8947, 8948, 8949, 8955, 8956, 8967, 8968, 8969, 8970, 8972, 8973, 8975, 8977, 8980, 8981, 8983, 8986, 8987, 8988, 8989, 8990, 8991, 9000, 9049, 9051, 9052, 9053, 9055, 9056, 9057, 9062, 9063, 9064, 9070, 9072, 9073, 9080, 9081, 9082, 9085, 9086, 9088, 9105, 9106, 9107, 9108, 9109, 9110, 9111, 9112, 9113, 9114, 9120, 9136, 9137, 9138, 9139, 9140, 9141, 9144, 9145, 9146, 9147, 9148, 9149, 9150, 9151, 9152, 9153, 9154, 9155, 9160, 9162, 9163, 9164, 9165, 9166, 9167, 9168, 9169, 9170, 9171, 9172, 9174, 9175, 9176, 9177, 9178, 9184, 9189, 9191, 9192, 9193, 9194, 9195, 9197, 9198, 9199, 9200, 9201, 9202, 9203, 9204, 9205, 9206, 9207, 9208, 9209, 9210, 9211, 9212, 9213, 9214, 9215, 9216, 9217, 9218, 9219, 9220, 9221, 9222, 9223, 9224, 9225, 9226, 9227, 9228, 9229, 9230, 9231, 9232, 9233, 9234, 9235, 9236, 9237, 9238, 9239, 9240, 9241, 9242, 9243, 9244, 9245, 9246, 9247, 9248, 9249, 9250, 9251, 9252, 9253, 9254, 9255, 9256, 9257, 9258, 9259, 9260, 9261, 9262, 9263, 9264, 9265, 9266, 9267, 9268, 9269, 9270, 9271, 9272, 9273, 9274, 9275, 9276, 9277, 9278, 9279, 9280, 9281, 9282, 9283, 9284, 9285, 9286, 9287, 9288, 9289, 9290, 9291, 9292, 9293, 9294, 9295, 9296, 9297, 9298, 9299, 9300, 9301, 9302, 9303, 9304, 9305, 9306, 9307, 9308, 9309, 9310, 9311, 9312, 9313, 9314, 9315, 9316, 9317, 9318, 9319, 9320, 9321, 9322, 9323, 9324, 9325, 9326, 9327, 9328, 9329, 9330, 9331, 9332, 9333, 9334, 9335, 9336, 9337, 9338, 9339, 9340, 9341, 9342, 9343, 9344, 9345, 9346, 9347, 9348, 9349, 9350, 9351, 9352, 9353, 9354, 9355, 9356, 9357, 9358, 9359, 9360, 9361, 9362, 9363, 9364, 9365, 9366, 9367, 9368, 9369, 9370, 9371, 9372, 9373, 9374, 9375, 9376, 9377, 9378, 9379, 9380, 9381, 9382, 9383, 9384, 9385, 9386, 9387, 9388, 9389, 9390, 9391, 9392, 9393, 9394, 9395, 9396, 9397, 9398, 9399, 9400, 9401, 9402, 9403, 9404, 9405, 9406, 9407, 9408, 9409, 9410, 9411, 9412, 9413, 9414, 9415, 9416, 9417, 9418, 9419, 9420, 9421, 9422, 9423, 9424, 9425, 9426, 9427, 9428, 9429, 9430, 9431, 9432, 9433, 9434, 9435, 9436, 9437, 9438, 9439, 9440, 9441, 9442, 9443, 9444, 9445, 9446, 9447, 9448, 9449, 9450, 9451, 9452, 9453, 9454, 9455, 9456, 9457, 9458, 9459, 9460, 9461, 9462, 9463, 9464, 9465, 9466, 9467, 9468, 9469, 9470, 9471, 9472, 9473, 9474, 9475, 9476, 9477, 9478, 9479, 9480, 9481, 9482, 9483, 9484, 9485, 9486, 9487, 9488, 9489, 9490, 9491, 9492, 9493, 9494, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9505, 9506, 9507, 9508, 9509, 9510, 9511, 9512, 9513, 9514, 9515, 9516, 9517, 9518, 9526, 9528, 9529, 9530, 9533, 9534, 9535, 9536, 9540, 9541, 9542, 9543, 9544, 9546, 9547, 9548, 9570, 9572, 9573, 9574, 9578, 9580, 9581, 9583, 9584, 9586, 9587, 9588, 9589, 9590, 9598, 9599, 9600, 9601, 9602, 9603, 9604, 9613, 9614, 9615, 9616, 9617, 9618, 9619, 9620, 9621, 9622, 9631, 9632, 9634, 9635, 9639, 9640, 9642, 9647, 9648, 9650, 9652, 9653, 9664, 9665, 9666, 9667, 9668, 9669, 9670, 9671, 9672, 9676, 9677, 9679, 9681, 9682, 9684, 9685, 9686, 9690, 9691, 9692, 9693, 9694, 9695, 9696, 9706, 9708, 9709, 9711, 9713, 9714, 9715, 9716, 9717, 9718, 9719, 9720, 9721, 9722, 9723, 9724, 9726, 9727, 9728, 9729, 9731, 9732, 9733, 9734, 9744, 9745, 9746, 9749, 9750, 9755, 9756, 9758, 9759, 9761, 9763, 9766, 9767, 9768, 9769, 9772, 9773, 9774, 9775, 9776, 9777, 9782, 9783, 9784, 9786, 9788, 9790, 9791, 9792, 9793, 9794, 9795, 9796, 9797, 9798, 9799, 9800, 9804, 9805, 9806, 9808, 9810, 9811, 9813, 9814, 9815, 9816, 9817, 9818, 9819, 9820, 9821, 9822, 9823, 9827, 9828, 9829, 9830, 9831, 9832, 9833, 9834, 9835, 9836, 9837, 9838, 9839, 9840, 9841, 9845, 9846, 9847, 9848, 9849, 9853, 9854, 9855, 9856, 9859, 9861, 9865, 9866, 9868, 9873, 9874, 9876, 9878, 9879, 9880, 9881, 9882, 9886, 9887, 9888, 9890, 9891, 9893, 9894, 9895, 9896, 9899, 9900, 9901, 9902, 9903, 9907, 9908, 9909, 9911, 9912, 9914, 9915, 9916, 9917, 9918, 9919, 9920, 9921, 9922, 9923, 9924, 9925, 9926, 9927, 9931, 9932, 9933, 9935, 9937, 9941, 9942, 9943, 9944, 9946, 9948, 9951, 9952, 9953, 9954, 9958, 9960, 9961, 9962, 9963, 9966, 9967, 9974, 9975, 9977, 9978, 9979, 9982, 9983, 9984, 9985, 9986, 9987, 9988, 9989, 9991, 9992, 9993, 9994, 9996, 9997, 9998, 9999, 10001, 10002, 10003, 10004, 10008, 10009, 10011, 10012, 10015, 10016, 10017, 10018, 10019, 10021, 10022, 10023, 10024, 10029, 10030, 10032, 10039, 10041, 10042, 10054, 10055, 10057, 10058, 10059, 10060, 10064, 10065, 10066, 10067, 10068, 10069, 10070, 10071, 10072, 10077, 10078, 10080, 10081, 10088, 10093, 10094, 10095, 10096, 10097, 10098, 10099, 10100, 10101, 10102, 10103, 10104, 10105, 10106, 10107, 10108, 10109, 10110, 10111, 10112, 10113, 10114, 10119, 10120, 10121, 10122, 10124, 10125, 10126, 10127, 10129, 10130, 10131, 10132, 10134, 10135, 10136, 10137, 10138, 10139, 10140, 10141, 10142, 10146, 10147, 10150, 10151, 10156, 10158, 10159, 10160, 10162, 10164, 10165, 10166, 10167, 10168, 10169, 10170, 10171, 10172, 10174, 10175, 10176, 10177, 10186, 10187, 10188, 10189, 10191, 10192, 10193, 10194, 10201, 10202, 10203, 10204, 10206, 10207, 10208, 10209, 10210, 10211, 10212, 10213, 10215, 10216, 10217, 10218, 10219, 10220, 10221, 10222, 10224, 10225, 10226, 10227, 10228, 10229, 10230, 10231, 10232, 10234, 10235, 10236, 10237, 10238, 10239, 10241, 10242, 10243, 10244, 10245, 10246, 10247, 10248, 10249, 10251, 10252, 10254, 10255, 10256, 10257, 10258, 10259, 10260, 10261, 10262, 10263, 10264, 10266, 10267, 10272, 10273, 10274, 10277, 10280, 10281, 10282, 10283, 10284, 10285, 10287, 10288, 10289, 10290, 10292, 10293, 10294, 10295, 10298, 10299, 10301, 10303, 10304, 10305, 10306, 10307, 10308, 10309, 10310, 10311, 10312, 10313, 10314, 10315, 10316, 10317, 10318, 10320, 10321, 10322, 10323, 10325, 10328, 10329, 10330, 10331, 10334, 10336, 10337, 10338, 10343, 10344, 10345, 10348, 10352, 10353, 10356, 10360, 10361, 10364, 10375, 10376, 10377, 10378, 10379, 10380, 10381, 10382, 10383, 10384, 10385, 10386, 10388, 10389, 10390, 10391, 10392, 10395, 10396, 10397, 10398, 10399, 10400, 10406, 10407, 10408, 10409, 10413, 10414, 10415, 10416, 10417, 10421, 10422, 10423, 10424, 10425, 10426, 10427, 10428, 10429, 10430, 10431, 10432, 10445, 10453, 10454, 10455, 10456, 10457, 10459, 10460, 10461, 10462, 10463, 10467, 10468, 10469, 10470, 10471, 10472, 10473, 10474, 10475, 10476, 10477, 10478, 10490, 10492, 10493, 10497, 10498, 10499, 10500, 10501, 10502, 10504, 10505, 10506, 10507, 10508, 10510, 10511, 10512, 10513, 10514, 10515, 10516, 10517, 10518, 10519, 10520, 10521, 10522, 10535, 10537, 10538, 10539, 10540, 10541, 10542, 10543, 10544, 10545, 10546, 10547, 10548, 10549, 10550, 10551, 10552, 10553, 10557, 10559, 10560, 10563, 10564, 10565, 10566, 10568, 10569, 10570, 10571, 10572, 10577, 10579, 10580, 10581, 10582, 10583, 10585, 10586, 10587, 10589, 10590, 10591, 10593, 10594, 10595, 10596, 10597, 10600, 10601, 10602, 10603, 10604, 10608, 10610, 10611, 10612, 10613, 10615, 10619, 10621, 10622, 10623, 10624, 10626, 10630, 10631, 10632, 10634, 10637, 10638, 10639, 10640, 10641, 10642, 10643, 10644, 10645, 10646, 10647, 10648, 10649, 10652, 10653, 10654, 10655, 10656, 10661, 10662, 10664, 10665, 10666, 10668, 10669, 10670, 10671, 10672, 10677, 10678, 10679, 10680, 10681, 10682, 10683, 10684, 10685, 10686, 10690, 10691, 10693, 10694, 10695, 10697, 10700, 10702, 10703, 10704, 10705, 10706, 10707, 10708, 10709, 10710, 10711, 10712, 10713, 10714, 10715, 10716, 10717, 10718, 10719, 10720, 10721, 10722, 10723, 10724, 10728, 10729, 10730, 10731, 10732, 10733, 10734, 10735, 10736, 10737, 10738, 10739, 10740, 10741, 10742, 10743, 10744, 10745, 10746, 10747, 10748, 10749, 10750, 10751, 10752, 10753, 10754, 10755, 10756, 10757, 10758, 10759, 10760, 10761, 10762, 10766, 10767, 10768, 10769, 10770, 10772, 10773, 10774, 10776, 10778, 10779, 10780, 10781, 10782, 10784, 10785, 10786, 10787, 10788, 10789, 10792, 10793, 10794, 10795, 10796, 10799, 10800, 10801, 10802, 10803, 10804, 10805, 10806, 10807, 10808, 10809, 10812, 10813, 10814, 10815, 10816, 10817, 10818, 10819, 10820, 10821, 10822, 10823, 10824, 10825, 10830, 10831, 10832, 10833, 10834, 10835, 10836, 10837, 10838, 10839, 10840, 10841, 10842, 10843, 10844, 10845, 10846, 10847, 10848, 10849, 10854, 10855, 10856, 10857, 10858, 10859, 10860, 10861, 10862, 10865, 10866, 10867, 10868, 10869, 10870, 10871, 10877, 10878, 10879, 10881, 10882, 10883, 10884, 10885, 10886, 10889, 10890, 10891, 10892, 10893, 10894, 10895, 10896, 10897, 10898, 10899, 10900, 10901, 10902, 10903, 10904, 10905, 10906, 10907, 10913, 10914, 10915, 10916, 10917, 10920, 10921, 10922, 10923, 10924, 10926, 10927, 10928, 10929, 10930, 10931, 10932, 10933, 10934, 10935, 10936, 10937, 10938, 10939, 10940, 10941, 10942, 10943, 10944, 10945, 10946, 10947, 10948, 10959, 10960, 10961, 10962, 10965, 10966, 10967, 10968, 10969, 10970, 10971, 10972, 10973, 10974, 10975, 10976, 10977, 10978, 10982, 10983, 10984, 10986, 10987, 10988, 10989, 10990, 10991, 10992, 10993, 10994, 10995, 11007, 11008, 11009, 11010, 11012, 11013, 11015, 11016, 11017, 11018, 11020, 11021, 11022, 11023, 11024, 11030, 11031, 11032, 11034, 11038, 11039, 11040, 11042, 11046, 11047, 11049, 11050, 11051, 11052, 11053, 11054, 11055, 11056, 11057, 11058, 11059, 11060, 11061, 11062, 11063, 11064, 11065, 11067, 11068, 11069, 11070, 11071, 11072, 11073, 11074, 11075, 11076, 11077, 11080, 11081, 11082, 11083, 11084, 11085, 11086, 11087, 11088, 11089, 11090, 11091, 11092, 11093, 11094, 11095, 11096, 11097, 11098, 11099, 11100, 11101, 11102, 11103, 11104, 11105, 11106, 11107, 11108, 11109, 11110, 11111, 11112, 11113, 11114, 11115, 11116, 11117, 11118, 11119, 11120, 11122, 11124, 11125, 11129, 11130, 11132, 11133, 11134, 11135, 11136, 11137, 11138, 11139, 11140, 11141, 11142, 11143, 11146, 11147, 11148, 11151, 11152, 11153, 11154, 11155, 11156, 11157, 11158, 11159, 11163, 11165, 11166, 11173, 11185, 11193, 12158, 12159, 12545, 12776, 12777, 12782, 12784, 12786, 12789, 12790, 12791, 12792, 12793, 12794, 12795, 12796, 12802, 12803, 12804, 12805, 12809, 12810, 12812, 12813, 12814, 12816, 12817, 12818, 12819, 12820, 12821, 12822, 12823, 12824, 12825, 12830, 12831, 12832, 12833, 12834, 12835, 12836, 12837, 12838, 12839, 12840, 12843, 12844, 12845, 12846, 12847, 12848, 12849, 12850, 12851, 12852, 12853, 12858, 12859, 12860, 12862, 12863, 12864, 12866, 12868, 12869, 12870, 12871, 12872, 12873, 12876, 12877, 12878, 12879, 12880, 12882, 12883, 12884, 12885, 12886, 12887, 12888, 12889, 12890, 12891, 12892, 12893, 12894, 12895, 12896, 12897, 12898, 12900, 12902, 12903, 12904, 12905, 12908, 12909, 12910, 12911, 12912, 12913, 12914, 12916, 12917, 12918, 12919, 12920, 12921, 12922, 12923, 12924, 12925, 12926, 12927, 12928, 12929, 12930, 12932, 12933, 12934, 12935, 12936, 12937, 12938, 12939, 12940, 12945, 12946, 12947, 12948, 12953, 12954, 12955, 12956, 12961, 12962, 12966, 12967, 12968, 12969, 12970, 12971, 12972, 12973, 12974, 12975, 12983, 12985, 12986, 12988, 12994, 12995, 12999, 13000, 13001, 13002, 13003, 13004, 13005, 13010, 13011, 13012, 13013, 13014, 13015, 13016, 13018, 13019, 13020, 13021, 13022, 13023, 13024, 13025, 13026, 13027, 13028, 13029, 13030, 13031, 13032, 13033, 13034, 13035, 13036, 13037, 13038, 13039, 13040, 13041, 13042, 13043, 13044, 13045, 13046, 13047, 13048, 13049, 13050, 13051, 13052, 13053, 13054, 13055, 13057, 13058, 13059, 13060, 13061, 13065, 13066, 13067, 13068, 13069, 13070, 13071, 13072, 13073, 13074, 13075, 13076, 13081, 13082, 13083, 13084, 13085, 13086, 13087, 13089, 13090, 13091, 13092, 13093, 13094, 13095, 13096, 13097, 13099, 13100, 13101, 13102, 13103, 13104, 13105, 13106, 13107, 13110, 13111, 13112, 13113, 13114, 13115, 13116, 13117, 13118, 13119, 13124, 13125, 13126, 13127, 13128, 13129, 13130, 13131, 13132, 13134, 13135, 13136, 13137, 13138, 13139, 13140, 13141, 13143, 13144, 13145, 13146, 13147, 13149, 13151, 13152, 13153, 13154, 13155, 13156, 13157, 13158, 13160, 13161, 13162, 13163, 13164, 13165, 13166, 13167, 13168, 13170, 13171, 13172, 13173, 13174, 13175, 13176, 13177, 13178, 13179, 13180, 13181, 13182, 13183, 13184, 13185, 13186, 13187, 13188, 13189, 13190, 13192, 13193, 13194, 13195, 13196, 13197, 13198, 13200, 13205, 13209, 13210, 13211, 13212, 13213, 13214, 13218, 13220, 13221, 13223, 13225, 13226, 13229, 13230, 13232, 13233, 13234, 13235, 13236, 13237, 13238, 13244, 13272, 13273, 13274, 13275, 13276, 13277, 13278, 13279, 13281, 13287, 13288, 13289, 13290, 13291, 13292, 13293, 13294, 13295, 13297, 13302, 13303, 13304, 13305, 13306, 13307, 13314, 13315, 13316, 13322, 13323, 13324, 13325, 13326, 13327, 13328, 13329, 13330, 13331, 13334, 13335, 13336, 13337, 13338, 13339, 13340, 13341, 13343, 13344, 13345, 13346, 13347, 13348, 13349, 13350, 13351, 13352, 13353, 13354, 13356, 13357, 13358, 13359, 13360, 13361, 13362, 13363, 13364, 13365, 13366, 13367, 13368, 13369, 13370, 13371, 13372, 13374, 13375, 13376, 13377, 13383, 13384, 13385, 13386, 13387, 13388, 13389, 13390, 13391, 13392, 13393, 13394, 13395, 13396, 13397, 13398, 13401, 13402, 13403, 13404, 13405, 13406, 13407, 13408, 13409, 13410, 13411, 13412, 13413, 13414, 13415, 13416, 13417, 13418, 13419, 13422, 13423, 13424, 13425, 13426, 13428, 13429, 13430, 13431, 13432, 13434, 13435, 13436, 13437, 13438, 13439, 13440, 13441, 13442, 13447, 13448, 13449, 13450, 13451, 13457, 13462, 13463, 13464, 13465, 13466, 13468, 13469, 13470, 13480, 13481, 13482, 13483, 13484, 13485, 13486, 13488, 13493, 13494, 13495, 13497, 13498, 13500, 13501, 13502, 13503, 13504, 13505, 13506, 13509, 13510, 13511, 13512, 13513, 13514, 13515, 13516, 13517, 13518, 13519, 13520, 13521, 13522, 13524, 13525, 13526, 13527, 13528, 13529, 13530, 13531, 13532, 13533, 13534, 13535, 13536, 13537, 13538, 13539, 13540, 13541, 13542, 13543, 13545, 13546, 13547, 13548, 13549, 13550, 13551, 13552, 13553, 13556, 13558, 13559, 13560, 13561, 13562, 13563, 13564, 13565, 13570, 13571, 13572, 13573, 13574, 13575, 13577, 13578, 13579, 13580, 13582, 13583, 13584, 13585, 13586, 13587, 13588, 13589, 13590, 13591, 13592, 13593, 13594, 13595, 13596, 13597, 13598, 13599, 13600, 13601, 13602, 13603, 13604, 13606, 13607, 13609, 13610, 13611, 13612, 13613, 13614, 13615, 13616, 13617, 13618, 13619, 13620, 13621, 13622, 13623, 13624, 13625, 13627, 13628, 13629, 13630, 13631, 13632, 13633, 13634, 13640, 13641, 13646, 13648, 13651, 13653, 13654, 13655, 13659, 13660, 13661, 13662, 13663, 13664, 13665, 13668, 13675, 13676, 13677, 13678, 13679, 13680, 13681, 13682, 13683, 13684, 13685, 13686, 13687, 13688, 13689, 13690, 13691, 13692, 13693, 13698, 13708, 13709, 13710, 13711, 13712, 13713, 13714, 13715, 13721, 13723, 13733, 13744, 13746, 13747, 13748, 13749, 13750, 13751, 13752, 13753, 13754, 13755, 13756, 13757, 13758, 13759, 13761, 13762, 13763, 13764, 13765, 13766, 13767, 13768, 13769, 13770, 13771, 13772, 13773, 13774, 13775, 13776, 13777, 13778, 13781, 13782, 13783, 13784, 13785, 13786, 13787, 13788, 13789, 13790, 13791, 13792, 13793, 13808, 13809, 13810, 13811, 13817, 13818, 13833, 13844, 13846, 13848, 13849, 13850, 13851, 13854, 13855, 13856, 13857, 13858, 13859, 13861, 13862, 13863, 13865, 13866, 13867, 13868, 13871, 13873, 13874, 13875, 13876, 13877, 13879, 13880, 13882, 13883, 13884, 13885, 13886, 13887, 13888, 13889, 13890, 13891, 13892, 13893, 13894, 13895, 13896, 13897, 13898, 13899, 13900, 13901, 13902, 13903, 13904, 13905, 13906, 13908, 13909, 13910, 13911, 13912, 13913, 13914, 13915, 13916, 13917, 13921, 13923, 13924, 13925, 13926, 13927, 13928, 13929, 13930, 13931, 13932, 13933, 13934, 13935, 13936, 13940, 13941, 13942, 13943, 13945, 13946, 13947, 13948, 13949, 13950, 13952, 13955, 13956, 13957, 13958, 13959, 13960, 13961, 13962, 13965, 13966, 13967, 13968, 13969, 13972, 13973, 13974, 13975, 13976, 13979, 13984, 13985, 13986, 13987, 13988, 13989, 13992, 13993, 13994, 13995, 13996, 13997, 13998, 13999, 14002, 14003, 14004, 14005, 14007, 14008, 14009, 14010, 14011, 14012, 14013, 14014, 14015, 14016, 14017, 14018, 14019, 14020, 14021, 14022, 14023, 14024, 14025, 14026, 14028, 14029, 14030, 14031, 14032, 14034, 14035, 14036, 14037, 14038, 14041, 14046, 14047, 14048, 14049, 14050, 14051, 14053, 14054, 14055, 14056, 14057, 14059, 14060, 14061, 14062, 14063, 14064, 14065, 14066, 14067, 14068, 14069, 14073, 14074, 14075, 14076, 14077, 14082, 14083, 14084, 14085, 14086, 14087, 14088, 14089, 14090, 14091, 14092, 14093, 14094, 14095, 14096, 14097, 14098, 14100, 14101, 14102, 14103, 14105, 14106, 14107, 14108, 14109, 14110, 14111, 14112, 14113, 14114, 14115, 14117, 14122, 14123, 14128, 14133, 14139, 14140, 14141, 14142, 14147, 14148, 14149, 14150, 14151, 14152, 14153, 14154, 14155, 14156, 14157, 14158, 14159, 14160, 14161, 14162, 14163, 14164, 14165, 14166, 14174, 14179, 14181, 14182, 14186, 14190, 14192, 14196, 14197, 14198, 14199, 14200, 14201, 14203, 14209, 14210, 14212, 14214, 14215, 14216, 14217, 14218, 14219, 14220, 14225, 14226, 14227, 14229, 14230, 14231, 14232, 14233, 14234, 14235, 14236, 14237, 14238, 14240, 14242, 14243, 14246, 14247, 14248, 14249, 14250, 14255, 14259, 14274, 14275, 14276, 14277, 14278, 14280, 14281, 14282, 14283, 14284, 14289, 14300, 14301, 14302, 14303, 14318, 14319, 14320, 14321, 14331, 14332, 14333, 14334, 14335, 14337, 14338, 14339, 14340, 14341, 14342, 14343, 14344, 14345, 14346, 14347, 14348, 14349, 14350, 14351, 14352, 14353, 14354, 14355, 14356, 14357, 14359, 14364, 14370, 14371, 14375, 14376, 14380, 14382, 14408, 14409, 14410, 14411, 14412, 14413, 14414, 14420, 14421, 14422, 14423, 14424, 14425, 14426, 14427, 14432, 14433, 14434, 14435, 14436, 14437, 14438, 14439, 14440, 14441, 14442, 14443, 14444, 14445, 14446, 14447, 14448, 14449, 14450, 14453, 14455, 14457, 14466, 14467, 14468, 14500, 14507, 14508, 14509, 14510, 14511, 14516, 14517, 14518, 14519, 14520, 14521, 14522, 14523, 14524, 14525, 14526, 14527, 14528, 14529, 14530, 14531, 14535, 14536, 14537, 14538, 14539, 14540, 14541, 14542, 14543, 14544, 14546, 14551, 14556, 14561, 14562, 14563, 14564, 14565, 14566, 14567, 14568, 14569, 14570, 14571, 14572, 14581, 14582, 14583, 14584, 14585, 14586, 14587, 14588, 14589, 14590, 14591, 14592, 14593, 14596, 14597, 14598, 14601, 14606, 14612, 14613, 14614, 14615, 14620, 14621, 14622, 14623, 14637, 14638, 14639, 14654, 14655, 14656, 14657, 14665, 14677, 14679, 14680, 14681, 14682, 14683, 14689, 14690, 14691, 14695, 14696, 14697, 14702, 14704, 14736, 14737, 14738, 14739, 14746, 14764, 14767, 14768, 14769, 14770, 14771, 14772, 14773, 14783, 14803, 14804, 14805, 14807, 14808, 14809, 14810, 14811, 14812, 14818, 14825, 14831, 14832, 14838, 14849, 14855, 14867, 14881, 14882, 14883, 14888, 14900, 14902, 14913, 14914, 14915, 14916, 14917, 14918, 14925, 14930, 14931, 14932, 14933, 14934, 14935, 14936, 14944, 14950, 14951, 14952, 14953, 14982, 14983, 14984, 15019, 15028, 15031, 15032, 15033, 15034, 15035, 15036, 15037, 15074, 15075, 15076, 15077, 15079, 15080, 15081, 15082, 15087, 15088, 15089, 15091, 15101, 15102, 15103, 15135, 15136, 15137, 15138, 15139, 15140, 15141, 15142, 15143, 15144, 15150, 15151, 15152, 15153, 15169, 15170, 15171, 15172, 15173, 15174, 15176, 15177, 15178, 15184, 15185, 15186, 15191, 15197, 15198, 15204, 15206, 15207, 15208, 15209, 15210, 15211, 15212, 15218, 15231, 15239, 15241, 15247, 15250, 15256, 15257, 15258, 15271, 15284, 15285, 15290, 15293, 15294, 15295, 15296, 15307, 15363, 15378, 15381, 15395, 15402, 15419, 15420, 15428, 15429, 15434, 15435, 15440, 15459, 15476, 15477, 15478, 15489, 15498, 15508, 15865, 16413, 16873, 17130, 17137, 17140, 17149, 17370, 17377, 17382, 17418, 17435, 17673, 17679, 17824, 17827, 18104, 18681, 19165, 19472, 19538, 19539, 19540, 19541, 19542, 19557, 19563]
[2009-11-20 03:00:07,611]   INFO -  jetbrains.buildServer.CLEANUP - Builds to clean for cleanup level (Clean artifacts):      762 builds: [1135, 1145, 1162, 1164, 1805, 1806, 1807, 1808, 1809, 1810, 1833, 1838, 1839, 1840, 1841, 1842, 1843, 1844, 1846, 1847, 1848, 1849, 1850, 1851, 1868, 1869, 7567, 7568, 9173, 9771, 9870, 9871, 9872, 9883, 9884, 9885, 9898, 9904, 9905, 9906, 9928, 9929, 9930, 9939, 9945, 9947, 9964, 9965, 9969, 9970, 9971, 9972, 9981, 10005, 10006, 10025, 10026, 10027, 10035, 10036, 10037, 10038, 10045, 10047, 10049, 10051, 10052, 10061, 10062, 10063, 10073, 10074, 10075, 10083, 10084, 10086, 10087, 10090, 10115, 10116, 10117, 10148, 10149, 10152, 10153, 10154, 10173, 10178, 10179, 10180, 10182, 10183, 10195, 10196, 10197, 10198, 10199, 10200, 10205, 10268, 10269, 10270, 10278, 10296, 10324, 10333, 10339, 10340, 10341, 10349, 10350, 10351, 10357, 10358, 10359, 10365, 10367, 10368, 10369, 10370, 10371, 10372, 10373, 10374, 10393, 10394, 10401, 10402, 10403, 10411, 10418, 10419, 10420, 10433, 10434, 10435, 10436, 10437, 10438, 10439, 10440, 10441, 10442, 10443, 10444, 10447, 10449, 10450, 10451, 10452, 10458, 10464, 10465, 10466, 10479, 10480, 10481, 10482, 10483, 10484, 10485, 10486, 10487, 10488, 10489, 10491, 10494, 10495, 10496, 10503, 10509, 10527, 10534, 10536, 10554, 10555, 10556, 10567, 10574, 10575, 10578, 10588, 10599, 10605, 10606, 10607, 10614, 10616, 10617, 10618, 10625, 10627, 10628, 10629, 10633, 10635, 10636, 10651, 10657, 10660, 10663, 10673, 10675, 10687, 10688, 10689, 10696, 10699, 10701, 10725, 10726, 10727, 10763, 10764, 10765, 10775, 10777, 10783, 10791, 10797, 10810, 10811, 10827, 10828, 10851, 10853, 10864, 10872, 10874, 10875, 10876, 10880, 10888, 10909, 10910, 10912, 10918, 10925, 10950, 10952, 10953, 10956, 10957, 10964, 10979, 10980, 10981, 10985, 10997, 10998, 11001, 11019, 11027, 11028, 11029, 11033, 11035, 11036, 11037, 11041, 11043, 11044, 11045, 11048, 11066, 11078, 11121, 11126, 11127, 11128, 11131, 11145, 11150, 11160, 11161, 11162, 11164, 11184, 11191, 11955, 11956, 11957, 12544, 12775, 12781, 12783, 12797, 12798, 12799, 12800, 12801, 12806, 12807, 12808, 12811, 12815, 12826, 12827, 12828, 12829, 12842, 12854, 12855, 12856, 12857, 12901, 12906, 12907, 12941, 12942, 12943, 12944, 12949, 12950, 12951, 12952, 12957, 12958, 12959, 12960, 12963, 12964, 12965, 12996, 12997, 12998, 13006, 13007, 13008, 13009, 13063, 13064, 13077, 13078, 13079, 13080, 13109, 13120, 13121, 13122, 13123, 13150, 13191, 13201, 13202, 13203, 13204, 13206, 13207, 13208, 13216, 13217, 13219, 13228, 13231, 13239, 13240, 13241, 13260, 13261, 13262, 13263, 13282, 13283, 13284, 13285, 13286, 13298, 13299, 13300, 13301, 13308, 13309, 13310, 13311, 13312, 13313, 13317, 13318, 13319, 13320, 13321, 13373, 13378, 13379, 13380, 13381, 13382, 13399, 13420, 13421, 13444, 13445, 13446, 13452, 13453, 13454, 13455, 13456, 13458, 13459, 13460, 13461, 13471, 13472, 13473, 13474, 13475, 13476, 13477, 13478, 13479, 13487, 13489, 13490, 13491, 13492, 13496, 13499, 13508, 13557, 13566, 13567, 13568, 13569, 13576, 13608, 13635, 13636, 13637, 13638, 13639, 13642, 13643, 13644, 13645, 13647, 13652, 13656, 13657, 13658, 13667, 13671, 13672, 13673, 13674, 13694, 13695, 13696, 13697, 13716, 13717, 13718, 13724, 13725, 13726, 13727, 13728, 13729, 13730, 13731, 13732, 13812, 13813, 13814, 13815, 13816, 13864, 13870, 13907, 13918, 13919, 13920, 13922, 13939, 13944, 13963, 13964, 13971, 13978, 13980, 13981, 13982, 13983, 13990, 14001, 14006, 14040, 14042, 14043, 14044, 14045, 14078, 14080, 14081, 14104, 14118, 14119, 14120, 14121, 14124, 14125, 14126, 14127, 14129, 14130, 14131, 14132, 14134, 14135, 14136, 14137, 14138, 14143, 14144, 14145, 14146, 14170, 14175, 14176, 14177, 14178, 14180, 14183, 14184, 14185, 14187, 14188, 14189, 14191, 14193, 14194, 14195, 14202, 14204, 14205, 14207, 14208, 14211, 14213, 14221, 14222, 14223, 14224, 14228, 14244, 14245, 14252, 14253, 14254, 14256, 14267, 14268, 14269, 14270, 14271, 14272, 14273, 14285, 14286, 14287, 14288, 14290, 14291, 14292, 14293, 14294, 14295, 14296, 14297, 14298, 14299, 14304, 14305, 14306, 14307, 14308, 14309, 14310, 14311, 14312, 14317, 14322, 14323, 14324, 14325, 14360, 14361, 14362, 14363, 14374, 14381, 14386, 14387, 14388, 14395, 14396, 14397, 14398, 14400, 14428, 14429, 14430, 14431, 14452, 14501, 14512, 14513, 14514, 14515, 14532, 14533, 14534, 14545, 14547, 14548, 14549, 14550, 14552, 14553, 14554, 14555, 14557, 14558, 14559, 14560, 14573, 14574, 14575, 14602, 14603, 14604, 14605, 14607, 14608, 14609, 14610, 14611, 14658, 14659, 14660, 14661, 14662, 14663, 14664, 14666, 14667, 14668, 14669, 14692, 14693, 14694, 14698, 14699, 14700, 14701, 14705, 14745, 14747, 14765, 14774, 14775, 14776, 14777, 14784, 14787, 14819, 14884, 14885, 14886, 14887, 14889, 14890, 14891, 14892, 14893, 14919, 14921, 14922, 14923, 14924, 14926, 14927, 14928, 14929, 14937, 14978, 15020, 15021, 15022, 15023, 15024, 15078, 15083, 15084, 15085, 15092, 15093, 15094, 15095, 15096, 15098, 15099, 15100, 15145, 15146, 15147, 15148, 15149, 15155, 15156, 15157, 15158, 15179, 15180, 15181, 15182, 15183, 15187, 15188, 15189, 15190, 15192, 15193, 15194, 15195, 15196, 15214, 15215, 15216, 15217, 15252, 15253, 15254, 15255, 15287, 15288, 15289, 15375, 15376, 15377, 15383, 15384, 15385, 15386, 15387, 15388, 15389, 15406, 15408, 15433, 15439, 19473, 19543, 19544, 19545, 19546, 19547, 19548, 19551, 19556, 19569]
[2009-11-20 03:00:07,642]   INFO -  jetbrains.buildServer.CLEANUP - Performing cleanup EVERYTHING for 3000 builds
[2009-11-20 03:00:07,642]   INFO -  jetbrains.buildServer.CLEANUP - Cleaning extensions data...
[2009-11-20 03:01:15,379]   INFO -  jetbrains.buildServer.CLEANUP - Deleting build logs...
[2009-11-20 03:01:27,708]   INFO -  jetbrains.buildServer.CLEANUP - Cleaning build history data from the database...
[2009-11-20 03:01:54,850]  ERROR -  jetbrains.buildServer.CLEANUP - Unexpected error occured during cleanup

org.springframework.transaction.TransactionSystemException: Could not roll back JDBC transaction; nested exception is java.sql.SQLException: Invalid state, the Connection object is closed.
     at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:279)
     at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:800)
     at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:777)
     at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:154)
     at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:132)
     at jetbrains.buildServer.serverSide.impl.DBFacade.compact(DBFacade.java:43)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:96)
     at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:5)
     at java.util.TimerThread.mainLoop(Timer.java:512)
     at java.util.TimerThread.run(Timer.java:462)
Caused by: java.sql.SQLException: Invalid state, the Connection object is closed.
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.checkOpen(ConnectionJDBC2.java:1634)
     at net.sourceforge.jtds.jdbc.ConnectionJDBC2.rollback(ConnectionJDBC2.java:2027)
     at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
     at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
     at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
     at org.springframework.jdbc.datasource.DataSourceTransactionManager.doRollback(DataSourceTransactionManager.java:276)
     ... 9 more
[2009-11-20 03:01:55,100]   INFO -   jetbrains.buildServer.SERVER - Optimizing search index...
[2009-11-20 03:01:55,178]   INFO -   jetbrains.buildServer.SERVER - Finished index optimization


Regards,
Thomas

0
Comment actions Permalink

Hi,

The same issue is happening again. TeamCity is not returning from maintenance mode and aws running for 10 hours. It is not coming back even I restart TeamCity service, and I have to restart the database instance again to make it works.

I have added the properties from your previous post and also removed "poolPreparedStatements" property.

I will do an update later and see if any failure in coming days.

Regards,
Thomas

0
Comment actions Permalink

Hi,

After changing the database.properties file, TeamCity completed the clean-up process and I don't have to restart the database instance everyday to make TeamCity back to live.
However, I still find some random errors from the log, which looks like conenction issue. I have attached the log file here.

Regards,
Thomas



Attachment(s):
TeamCity.Log.2009-11-24.zip
0
Comment actions Permalink

Most of the errors in the log are network errors, i.e. TeamCity tries to connect to the server but server refuses connection.
Also it seems database server sometimes works in single user mode:
Login failed for user '****'. Reason: Server is in single user mode. Only one administrator can connect at this time.

Are you sure you do not have some periodic task making DB backup or something like this which locks the database or somehow limits the number of possible connections?

0

Please sign in to leave a comment.