Teamcity keeps hanging when manually starting a build

Hi.

Running 9.1.1 (build 37059) on Ubuntu 15.04.

When I use the web interface to start a build, the UI becomes unresponsive, and remains so after refreshing the page etc.

In the server log, I see the following:

[2015-10-20 09:03:58,048]   INFO - tbrains.buildServer.ACTIVITIES - "..." build configuration settings were edited ("version before: 4, version after: 5") by "..." with comment "general settings of '...' build configuration were updated"
[2015-10-20 09:04:13,867]   INFO - ity.BuildQueuePriorityOrdering - New item Queued build {Build promotion {...} with weight 0.00 inserted at the default position 0 in the end of the queue
[2015-10-20 09:04:13,930]   INFO - tbrains.buildServer.ACTIVITIES - Build BUILD_PROMOTION{id=211} was added to queue by ...
[2015-10-20 09:05:11,722]   INFO - iagnostic.web.DiagnosticFilter - Thread dump automatically saved to file "/opt/jetbrains/TeamCity/logs/threadDumps-2015-10-20/2015-10-20_09.05.11-longRequest-1.txt" with summary: Request processing time (40000 ms) exceeded for the following 2 threads
[2015-10-20 09:05:31,733]   INFO - iagnostic.web.DiagnosticFilter - Thread dump automatically saved to file "/opt/jetbrains/TeamCity/logs/threadDumps-2015-10-20/2015-10-20_09.05.31-longRequest-1.txt" with summary: Request processing time (40000 ms) exceeded for the following 3 threads
...

In the thread dumps, I see several threads with this stack trace:

"TC: 09:07:24 /viewType.html?buildTypeId=...&__fragmentId=buildConfigurationContainerInner; IP: 85.119.25.68; http-nio-8111-exec-5" daemon group="main" prio=5 tid=86

nid=86 blocked on java.lang.Object@705dc3a8

   by "TC: 09:04:13 /ajax.html; POST; IP: 85.119.25.68; http-nio-8111-exec-7"

   java.lang.Thread.State: BLOCKED

    at jetbrains.buildServer.serverSide.impl.BuildPromotionImpl.getAllAuditLogActions(BuildPromotionImpl.java:1046)

    at jetbrains.buildServer.serverSide.impl.BuildPromotionImpl.getAuditLogActions(BuildPromotionImpl.java:999)

    at jetbrains.buildServer.serverSide.impl.UserActionsProvider.getIconUserActions(UserActionsProvider.java:13)

    at org.apache.jsp.tag.webpromotionCommentIcon_tag$Helper.invoke0(webpromotionCommentIcon_tag.java:354)

    at org.apache.jsp.tag.webpromotionCommentIcon_tag$Helper.invoke(webpromotionCommentIcon_tag.java:379)

    at org.apache.jsp.tag.webtrimWhitespace_tag._jspx_meth_c_005fset_005f0(webtrimWhitespace_tag.java:112)

    at org.apache.jsp.tag.webtrimWhitespace_tag.doTag(webtrimWhitespace_tag.java:71)

    at org.apache.jsp.tag.webpromotionCommentIcon_tag.doTag(webpromotionCommentIcon_tag.java:125)

    at org.apache.jsp.tag.webbuildCommentIcon_tag._jspx_meth_bs_005fpromotionCommentIcon_005f0(webbuildCommentIcon_tag.java:120)

    at org.apache.jsp.tag.webbuildCommentIcon_tag.doTag(webbuildCommentIcon_tag.java:93)

    at org.apache.jsp.tag.webbuildRow_tag._jspx_meth_bs_005fbuildCommentIcon_005f0(webbuildRow_tag.java:1796)

    at org.apache.jsp.tag.webbuildRow_tag._jspx_meth_c_005fif_005f7(webbuildRow_tag.java:1713)

    at org.apache.jsp.tag.webbuildRow_tag.doTag(webbuildRow_tag.java:517)

    at org.apache.jsp.buildTypeStatusDiv_jsp._jspx_meth_bs_005fbuildRow_005f0(buildTypeStatusDiv_jsp.java:877)

    at org.apache.jsp.buildTypeStatusDiv_jsp._jspx_meth_c_005fforEach_005f0(buildTypeStatusDiv_jsp.java:824)

    at org.apache.jsp.buildTypeStatusDiv_jsp._jspx_meth_c_005fif_005f3(buildTypeStatusDiv_jsp.java:785)

    at org.apache.jsp.buildTypeStatusDiv_jsp._jspService(buildTypeStatusDiv_jsp.java:419)

    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)

    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)

    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)

    at jetbrains.spring.web.TeamCityJSPServletImpl.access$001(TeamCityJSPServletImpl.java:30)

    at jetbrains.spring.web.TeamCityJSPServletImpl$2.apply(TeamCityJSPServletImpl.java)

    at jetbrains.buildServer.util.Util.doUnderContextClassLoader(Util.java:78)

    at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:28)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at jetbrains.spring.web.JspController.doHandle(JspController.java:9)

    at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:78)

    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:154)

    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)

    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)

    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)

    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)

    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)

    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:34)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)

    at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:604)

    at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:543)

    at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954)

    at org.apache.jsp.tag.webext.includeExtension_tag._jspx_meth_c_005fif_005f3(includeExtension_tag.java:521)

    at org.apache.jsp.tag.webext.includeExtension_tag._jspx_meth_c_005fif_005f0(includeExtension_tag.java:184)

    at org.apache.jsp.tag.webext.includeExtension_tag.doTag(includeExtension_tag.java:148)

    at org.apache.jsp.tag.webext.showTabs_tag._jspx_meth_ext_005fincludeExtension_005f0(showTabs_tag.java:632)

    at org.apache.jsp.tag.webext.showTabs_tag._jspx_meth_c_005fif_005f5(showTabs_tag.java:607)

    at org.apache.jsp.tag.webext.showTabs_tag.doTag(showTabs_tag.java:232)

    at org.apache.jsp.viewType_jsp$Helper.invoke26(viewType_jsp.java:2146)

    at org.apache.jsp.viewType_jsp$Helper.invoke(viewType_jsp.java:2322)

    at org.apache.jsp.tag.webrefreshable_tag.doTag(webrefreshable_tag.java:146)

    at org.apache.jsp.viewType_jsp$Helper.invoke19(viewType_jsp.java:2051)

    at org.apache.jsp.viewType_jsp$Helper.invoke(viewType_jsp.java:2301)

    at org.apache.jsp.tag.webpage_tag.doTag(webpage_tag.java:569)

    at org.apache.jsp.viewType_jsp._jspService(viewType_jsp.java:260)

    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)

    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)

    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)

    at jetbrains.spring.web.TeamCityJSPServletImpl.access$001(TeamCityJSPServletImpl.java:30)

    at jetbrains.spring.web.TeamCityJSPServletImpl$2.apply(TeamCityJSPServletImpl.java)

    at jetbrains.buildServer.util.Util.doUnderContextClassLoader(Util.java:78)

    at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:28)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at jetbrains.spring.web.JspController.doHandle(JspController.java:9)

    at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:78)

    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:154)

    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)

    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)

    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)

    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)

    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)

    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:34)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748)

    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486)

    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411)

    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338)

    at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:209)

    at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:267)

    at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1221)

    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1005)

    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:952)

    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)

    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)

    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)

    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:34)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:1)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:10)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:63)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:23)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)

    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)

    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)

    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)

    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

    at java.lang.Thread.run(Thread.java:745)

Can anyone shed any light as to what might be causing this?

Thanks,

Nick

1 comment
Comment actions Permalink

Hi Nick,

Could you please attach several full thread dumps?

0

Please sign in to leave a comment.