TeamCity server crash with too many open files error

Hi
I am using TeamCity 6.5.1 version and we experienced an outage on teamcity server.
We have deployed TeamCity on linux environment (

Linux  version 2.6.9-89.0.9.ELlargesmp (mockbuild@x86-002.build.bos.redhat.com)  (gcc version 3.4.6 20060404 (Red Hat 3.4.6-11)) #1 SMP Wed Aug 19 08:12:11 EDT  2009)


We got error "java.net.SocketException: Too many open files" and then the server crashed . After server crash we were able to login to teamcity site but we could not see any project there and hence could not do any normal build activity. The issue was resolved after restarting the server. Can you please suggest possible cause of the error and what can be done to avoid the error in future.
Please see below error in detail.


Regards
Ruby


Error from lof :



WARN -   jetbrains.buildServer.SERVER - Failed to obtain network addresses
java.net.SocketException: Too many open files
    at java.net.NetworkInterface.getAll(Native Method)
    at java.net.NetworkInterface.getNetworkInterfaces(NetworkInterface.java:286)
    at jetbrains.buildServer.NetworkUtil.getSelfAddresses(NetworkUtil.java:37)
    at jetbrains.buildServer.serverSide.impl.BuildAgentManagerImpl.isLocalAddress(BuildAgentManagerImpl.java:165)
    at jetbrains.buildServer.serverSide.impl.BuildAgentManagerImpl.registerAgentImpl(BuildAgentManagerImpl.java:57)
    at jetbrains.buildServer.serverSide.impl.BuildAgentManagerImpl.registerAgent(BuildAgentManagerImpl.java:278)
    at jetbrains.buildServer.serverSide.impl.auth.SecuredBuildAgentManager.registerAgent(SecuredBuildAgentManager.java:5)
    at jetbrains.buildServer.serverSide.impl.XmlRpcBasedServer.doRegisterAgent(XmlRpcBasedServer.java:167)
    at jetbrains.buildServer.serverSide.impl.XmlRpcBasedServer.registerAgent3(XmlRpcBasedServer.java:88)
    at sun.reflect.GeneratedMethodAccessor2031.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at jetbrains.buildServer.serverSide.impl.ExceptionCollectorWrapper.execute(ExceptionCollectorWrapper.java:56)
    at jetbrains.buildServer.controllers.XmlRpcController$2.execute(XmlRpcController.java:3)
    at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
    at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
    at jetbrains.buildServer.controllers.XmlRpcController$1$1.execute(XmlRpcController.java:3)
    at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
    at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:114)
    at jetbrains.buildServer.controllers.XmlRpcController$3.apply(XmlRpcController.java:1)
    at jetbrains.buildServer.controllers.XmlRpcController$3.apply(XmlRpcController.java:7)
    at jetbrains.buildServer.serverSide.impl.XmlRpcSessionManager.executeRequest(XmlRpcSessionManager.java:183)
    at jetbrains.buildServer.controllers.XmlRpcController.handleRequestInternal(XmlRpcController.java:11)
    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:95)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at jetbrains.buildServer.web.DiagnosticFilter.runChainWithModifiedThreadName(DiagnosticFilter.java:4)
    at jetbrains.buildServer.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:15)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:662)
1 comment
Comment actions Permalink

Hi Ruby

OS sets limits for a number of open handles on system-wide level, and separatelly for each process.
You'll need to check current limits by ulimit command, and a number of cuttently open files for Tomcat process by lsof command.
I found a good explanation in this blog post.

Limits can be increased. But if you suspect there is a leak at TeamCity side, please send us the logs and lsof output.

Michael

0

Please sign in to leave a comment.