TeamCity 3.1.1 resopnse on http requests fails after upgraiding

After upgrading from 3.0.1 Teamcity response on http request fails from
time to time, or takes to much time.

root@builder:/var/tainted# cat TeamCity/logs/teamcity-server.log

WARN - jetbrains.buildServer.SERVER -
Exception occured while communicating by XML RPC: ClientAbortException:
java.net.SocketException: Broken pipe /remote: 192.168.10.101:60043
ERROR - jetbrains.buildServer.SERVER -
Error in JSP, request dump:
Path: /overview.jsp
Method: GET


java.lang.OutOfMemoryError: Java heap space
ERROR - jetbrains.buildServer.SERVER -
Error in JSP, request dump:
Path: /runtimeError.jsp
Method: GET


java.lang.OutOfMemoryError: Java heap space
INFO - rs.login.LoginSubmitController -
Successful login with login name: online\peter.kamyshnikov
INFO - jetbrains.buildServer.SERVER -
Agent resbuilder has been unregistered. Id: 12
INFO - jetbrains.buildServer.SERVER -
Agent build-agent3 has been registered with id 1 currently runnung build -1
WARN - rverSide.impl.XmlRpcBasedAgent -
Problem pinging build-agent4 at http://192.168.10.170:9090. Firewall
problem? Or wrong IP of agent?
WARN - jetbrains.buildServer.SERVER -
Exception occured while communicating by XML RPC:
java.lang.OutOfMemoryError: Java heap space /remote: 192.168.10.170:35343
INFO - rs.login.LoginSubmitController -
Successful login with login name: online\dmitry.uvarkov
WARN - jetbrains.buildServer.SERVER -
ClientAbortException: java.net.SocketException: Connection reset
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:319)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at
org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at org.jdom.output.XMLOutputter.output(XMLOutputter.java:429)
at
jetbrains.buildServer.controllers.XmlResponseUtil.writeXmlResponse(XmlResponseUtil.java:1)
at
jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:12)
at
jetbrains.buildServer.controllers.BaseFormXmlController.doHandle(BaseFormXmlController.java:20)
at
jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:40)
at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:874)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:808)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:476)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:441)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:7)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: Connection reset
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:737)
at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at
org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:299)
at
org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:963)
at org.apache.coyote.Response.action(Response.java:183)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:314)
... 33 more
WARN - jetbrains.buildServer.SERVER -
Exception occured while communicating by XML RPC:
java.lang.OutOfMemoryError: Java heap space /remote: 192.168.10.101:54190
INFO - jetbrains.buildServer.SERVER -
Agent resbuilder has been registered with id 12 currently runnung build -1
INFO - rs.login.LoginSubmitController -
Successful login with login name: online\maria.kazarnovskaya
ERROR - jetbrains.buildServer.SERVER -
Error in JSP, request dump:
Path: /runtimeError.jsp
Method: POST


java.lang.OutOfMemoryError: Java heap space
ERROR - jetbrains.buildServer.SERVER -

java.lang.OutOfMemoryError: Java heap space

4 comments

The log you provided clearly indicates that out of memory error occurs (most likely on the server side). Take a look at this document: http://www.jetbrains.net/confluence/display/TCD3/Reporting+Issues#ReportingIssues-oom for details how to debug what causes this error.

--
Pavel Sher

0

Yes, Pavel, it is clear - I suppose, Vitaly suspected some configuration errors that could have caused this.
If it is a valid memory problem, I have another question: is it possible to (temporarily) disable some features of Teamcity 3.1.1. to decrease server memory demands?

Edited by: Peter Ivanov on Jun 16, 2008 4:50 PM

0

Well, basically we are trying to reduce memory usage in the newer versions and in 3.1.1 we've fixed potential memory leak so this new version should require less memory than 3.0.1. So it does not look like a valid error. I would suggest to launch server with -XX:+HeapDumpOnOutOfMemoryError and obtain memory snapshot if OOE occurs again. You can upload this snapshot to our FTP server: ftp://ftp.intellij.net/.uploads and send link to it to teamcity-feedback@jetbrains.com.

Please send us teamcity-server.log too and provide some details about your installation.

--
Pavel Sher

0

-Xmx512m fixed the problem, it seems.
I wonder why wasn't this parameter passed to Teamcity server initially, but it still might be some installation bug. If OOME occurs again, though, we will send you the snapshot.

Thank you.

0

Please sign in to leave a comment.