TeamCity 6.5.6 continuously restarts

Since last Friday, we have seen our server experiencing repeated restarts. My first clue came when I tried to bring up a log from a recent build, and saw this error:

Failed to read from messages file: /teamcity/.BuildServer/system/messages/CH93/297793.msg5": java.io.UTFDataFormatException: malformed input around byte 136




The restart condition does not show in the UI as the server being in the "Initialization" phase, but there is evidence in the catalina.out logs. I did a search in each of the previous 7 days' log, like this:
grep 'Server startup' catalina.<date>.log > <day>_starts and then wc -l <day>_starts

and summarize results here:
Day number of starts in log
Wednesday, 3/20 6
Thursday, 3/21 0
Friday, 3/22 193
Saturday, 3/23 360
Sunday, 3/24 360
Monday, 3/25 360
Tuesday, 3/26 357
Wednesday, 3/27 255 (as of 5:00PM)


Is this a known issue? Here are some relevant details:
Linux (CentOS 5.2) on VM with 4Gb RAM
TeamCity 6.5.6 (Build 18130)

1 comment
Comment actions Permalink

Additional detail from log:

Mar 28, 2013 12:01:20 AM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8111
Mar 28, 2013 12:05:15 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/jdk1.6.0_11/jre/lib/i386/server:/usr/java/jdk1.6.0_11/jre/lib/i386:/usr/java/jdk1.6.0_11/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib
Mar 28, 2013 12:05:15 AM org.apache.coyote.http11.Http11Protocol init
SEVERE: Error initializing endpoint
java.net.BindException: Address already in use <null>:8111
 at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:549)
 at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:176)
 at org.apache.catalina.connector.Connector.initialize(Connector.java:1014)
 at org.apache.catalina.core.StandardService.initialize(StandardService.java:680)
 at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:795)
 at org.apache.catalina.startup.Catalina.load(Catalina.java:524)
 at org.apache.catalina.startup.Catalina.load(Catalina.java:548)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Caused by: java.net.BindException: Address already in use
 at java.net.PlainSocketImpl.socketBind(Native Method)
 at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
 at java.net.ServerSocket.bind(ServerSocket.java:319)
 at java.net.ServerSocket.<init>(ServerSocket.java:185)
 at java.net.ServerSocket.<init>(ServerSocket.java:141)
 at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:50)
 at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
 ... 12 more
Mar 28, 2013 12:05:15 AM org.apache.catalina.startup.Catalina load
SEVERE: Catalina.start
LifecycleException:  Protocol handler initialization failed: java.net.BindException: Address already in use <null>:8111
 at org.apache.catalina.connector.Connector.initialize(Connector.java:1016)
 at org.apache.catalina.core.StandardService.initialize(StandardService.java:680)
 at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:795)
 at org.apache.catalina.startup.Catalina.load(Catalina.java:524)
 at org.apache.catalina.startup.Catalina.load(Catalina.java:548)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
 at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Mar 28, 2013 12:05:15 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 887 ms
Mar 28, 2013 12:05:15 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Mar 28, 2013 12:05:15 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.29
Mar 28, 2013 12:05:15 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor ROOT.xml

0

Please sign in to leave a comment.