java.lang.OutOfMemoryError: Java heap space Exception after REST API Call

Avatar
Permanently deleted user
Created

Hi,

our Teamcity Server was suffering from to less Memory and we got an error message pointing out that much CPU was used for memory cleaning.

As our Server still has RAM available I increased the XMX Setting to -Xmx1024m.

Now everything seems to be working fine and the Diagnostic pages says the following:

Total heap: 629.97 MB (68.4% of maximum available 920.50 MB)

Data: 592.78 MB (86.8% of maximum available 682.75 MB)

Our Server is using only the half of the available Ram. 

Nevertheless I'm getting the following Exception when performing a HTTP POST Request to add Artifacts via the Teamcity API.

 

Thanks in advance!  

 


Caused by: java.lang.OutOfMemoryError: Java heap space
[2018-02-02 11:13:25,575]   WARN [io-443-exec-108] - est.jersey.ExceptionMapperUtil - Error has occurred during request processing, status code: 500 (Internal Server Error). Details: javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space Request: POST '/app/rest/latest/buildTypes/id:Releases_VmaxRelease_1712_BuildRelease/artifact-dependencies', from client IPADDRESS:Port, user-agent "PostmanRuntime/7.1.1", authenticated as 'xxxxxx' {id=58}. 
javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:927)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
	at jetbrains.buildServer.server.rest.APIController$4.apply(APIController.java:471)
	at jetbrains.buildServer.server.rest.APIController$4.apply(APIController.java:457)
	at jetbrains.buildServer.util.Util.doUnderContextClassLoader(Util.java:78)
	at jetbrains.buildServer.server.rest.APIController.lambda$doHandle$2(APIController.java:457)
	at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadNameFuncThrow(NamedThreadFactory.java:102)
	at jetbrains.buildServer.server.rest.APIController.doHandle(APIController.java:455)
	at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:75)
	at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
	at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:34)
	at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:35)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:56)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:3)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
	at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:51)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:2)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at jetbrains.buildServer.web.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:7)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at jetbrains.buildServer.web.CSRFFilter.doFilter(CSRFFilter.java:100)
	at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
	at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
	at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:37)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:1)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space

Votes

0

Share

2 comments
Avatar
Denis Lapuente
Created

Hi Sven,

An Out of Memory error is more likely to happen during peaks in processing, so using over 60% on normal usage might be an indication that your server might get over the limit under specific circumstances. If under pressure you can see that the memory usage gets very high, over 80 or 90%, I'd recommend increasing the amount of memory: https://confluence.jetbrains.com/display/TCD10/Installing+and+Configuring+the+TeamCity+Server#InstallingandConfiguringtheTeamCityServer-SettingUpMemorysettingsforTeamCityServer

Also keep in mind the note mentioned there, if you have to go over 1 GB, it's recommended to use the 64 bit JVM and then double the amount.

Hope this helps.

0
Avatar
Sreepathy
Created

Usually, OutOfMemoryError occurs because of insufficient Java heap memory allocation, which indicates that your JVM is running out of memory to allocate for objects in the heap.  Even though you've allocated some GB to the container, there could be a possibility that the actual Java heap size being used by the JVM is much lower. This is because, always, JVM by default does not utilize the full container memory unless we instruct to.

In this case, the TeamCity server experienced memory pressure. This initially seems to be leading to excessive CPU usage during garbage collection. After increasing the heap size, the server appears to be performing better, with the diagnostic page showing sufficient available memory. But still you can see, the OutOfMemoryError triggered during the HTTP POST request to add artifacts via the TeamCity API suggests that a specific operation is consuming more memory than expected. When large data objects are processed all together, then this can cause issues. Also see if there is insufficient heap allocation for that particular workload. To overcome this issue, you can consider further increasing the heap size.

0

Please sign in to leave a comment.