High CPU usage

We have noticed the CPU usage of TeamCity increased rather substantially in the last week and it looks to have started at midnight on July the 1st for some reason.

DevLoadAvg.jpg

$ ps -ea -o pcpu,cmd | grep teamcity
246 /usr/bin/java -Djava.util.logging.config.file=/opt/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xmx512m -XX:MaxPermSize=270m -Dlog4j.configuration=file:/opt/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djava.awt.headless=true -Djava.endorsed.dirs=/opt/TeamCity/endorsed -classpath /opt/TeamCity/bin/bootstrap.jar:/opt/TeamCity/bin/tomcat-juli.jar -Dcatalina.base=/opt/TeamCity -Dcatalina.home=/opt/TeamCity -Djava.io.tmpdir=/opt/TeamCity/temp org.apache.catalina.startup.Bootstrap start



We had been running 7.0.2 and the HSQLDB so we have migrated to a local MySQL database and upgraded to 7.0.3 (after reading through http://youtrack.jetbrains.com/issue/TW-21375)

We also upgraded java to version 1.7.0 incase there was any issues with the java release.

# /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java -version
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.9) (rhel-1.36.b17.el6_0-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

# /usr/java/jdk1.7.0_05/bin/java -version
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.1-b03, mixed mode)


None of these have resolved the problem so its still sitting there sucking CPU time.

Are there any further solutions that we could try?
9 comments
Comment actions Permalink

Hi Christopher

Could you please capture CPU profiling snapshot as explained in Server Performance article.

Thanks

0
Comment actions Permalink

Just tried to install the performace profiler but the server refused to start when adding the '-agentpath:/opt/TeamCity/bin/libyjpagent.so=disablecounts,disablealloc,disablestacktelemetry,disableexceptiontelemetry,sessionname=TeamCity' option to the startup arguments and I get this error when I take it out and start it.



Error message: org.apache.jasper.JasperException: /plugins/serverProfile/profControls.jsp (line: 41, column: 73) Attribute className invalid for tag saving according to TLD
TeamCity: 7.0.3 (build 21424)
Operating system: Linux (2.6.32-71.29.1.el6.x86_64, amd64)
Java: 1.7.0_05-b05 (Oracle Corporation)
Servlet container: Apache Tomcat/7.0.23

Trace: javax.servlet.jsp.JspException: org.apache.jasper.JasperException: /plugins/serverProfile/profControls.jsp (line: 41, column: 73) Attribute className invalid for tag saving according to TLD
     at org.apache.jsp.tag.web.ext.includeExtension_tag.doTag(includeExtension_tag.java:96)
     at org.apache.jsp.tag.web.ext.includeExtensions_tag._jspx_meth_ext_005fincludeExtension_005f0(includeExtensions_tag.java:322)
     at org.apache.jsp.tag.web.ext.includeExtensions_tag._jspx_meth_c_005fforEach_005f2(includeExtensions_tag.java:291)
     at org.apache.jsp.tag.web.ext.includeExtensions_tag.doTag(includeExtensions_tag.java:134)
     at org.apache.jsp.admin.serverConfigGeneral_jsp._jspService(serverConfigGeneral_jsp.java:213)
     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
     at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
     at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
     at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:7)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.spring.web.JspController.doHandle(JspController.java:13)
     at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:74)
     at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
     at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
     at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
     at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954)
     at org.apache.jsp.tag.web.ext.includeExtension_tag._jspx_meth_c_005fif_005f0(includeExtension_tag.java:122)
     at org.apache.jsp.tag.web.ext.includeExtension_tag.doTag(includeExtension_tag.java:85)
     at org.apache.jsp.admin.adminMain_jsp._jspx_meth_ext_005fincludeExtension_005f0(adminMain_jsp.java:223)
     at org.apache.jsp.admin.adminMain_jsp.access$4(adminMain_jsp.java:212)
     at org.apache.jsp.admin.adminMain_jsp$Helper.invoke3(adminMain_jsp.java:313)
     at org.apache.jsp.admin.adminMain_jsp$Helper.invoke(adminMain_jsp.java:343)
     at org.apache.jsp.tag.web.page_tag.doTag(page_tag.java:670)
     at org.apache.jsp.admin.adminMain_jsp._jspx_meth_bs_005fpage_005f0(adminMain_jsp.java:138)
     at org.apache.jsp.admin.adminMain_jsp._jspService(adminMain_jsp.java:108)
     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
     at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
     at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
     at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:7)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.spring.web.JspController.doHandle(JspController.java:13)
     at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:74)
     at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
     at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
     at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
     at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
     at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:238)
     at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
     at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:6)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at jetbrains.buildServer.web.DiagnosticFilter.runChainWithModifiedThreadName(DiagnosticFilter.java:61)
     at jetbrains.buildServer.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:12)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:4)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
     at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.jasper.JasperException: /plugins/serverProfile/profControls.jsp (line: 41, column: 73) Attribute className invalid for tag saving according to TLD
     at org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:42)
     at org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:408)
     at org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:237)
     at org.apache.jasper.compiler.Validator$ValidateVisitor.checkXmlAttributes(Validator.java:1260)
     at org.apache.jasper.compiler.Validator$ValidateVisitor.visit(Validator.java:870)
     at org.apache.jasper.compiler.Node$CustomTag.accept(Node.java:1539)
     at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2376)
     at org.apache.jasper.compiler.Node$Visitor.visitBody(Node.java:2428)
     at org.apache.jasper.compiler.Validator$ValidateVisitor.visit(Validator.java:889)
     at org.apache.jasper.compiler.Node$CustomTag.accept(Node.java:1539)
     at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2376)
     at org.apache.jasper.compiler.Node$Visitor.visitBody(Node.java:2428)
     at org.apache.jasper.compiler.Node$Visitor.visit(Node.java:2434)
     at org.apache.jasper.compiler.Node$Root.accept(Node.java:475)
     at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2376)
     at org.apache.jasper.compiler.Validator.validateExDirectives(Validator.java:1790)
     at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:217)
     at org.apache.jasper.compiler.Compiler.compile(Compiler.java:373)
     at org.apache.jasper.compiler.Compiler.compile(Compiler.java:353)
     at org.apache.jasper.compiler.Compiler.compile(Compiler.java:340)
     at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:646)
     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:357)
     at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
     at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
     at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:7)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.spring.web.JspController.doHandle(JspController.java:13)
     at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:74)
     at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
     at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
     at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
     at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954)
     at org.apache.jsp.tag.web.ext.includeExtension_tag._jspx_meth_c_005fif_005f0(includeExtension_tag.java:122)
     at org.apache.jsp.tag.web.ext.includeExtension_tag.doTag(includeExtension_tag.java:85)
     ... 95 more
org.apache.jasper.JasperException: /plugins/serverProfile/profControls.jsp (line: 41, column: 73) Attribute className invalid for tag saving according to TLD
     at org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:42)
     at org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:408)
     at org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:237)
     at org.apache.jasper.compiler.Validator$ValidateVisitor.checkXmlAttributes(Validator.java:1260)
     at org.apache.jasper.compiler.Validator$ValidateVisitor.visit(Validator.java:870)
     at org.apache.jasper.compiler.Node$CustomTag.accept(Node.java:1539)
     at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2376)
     at org.apache.jasper.compiler.Node$Visitor.visitBody(Node.java:2428)
     at org.apache.jasper.compiler.Validator$ValidateVisitor.visit(Validator.java:889)
     at org.apache.jasper.compiler.Node$CustomTag.accept(Node.java:1539)
     at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2376)
     at org.apache.jasper.compiler.Node$Visitor.visitBody(Node.java:2428)
     at org.apache.jasper.compiler.Node$Visitor.visit(Node.java:2434)
     at org.apache.jasper.compiler.Node$Root.accept(Node.java:475)
     at org.apache.jasper.compiler.Node$Nodes.visit(Node.java:2376)
     at org.apache.jasper.compiler.Validator.validateExDirectives(Validator.java:1790)
     at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:217)
     at org.apache.jasper.compiler.Compiler.compile(Compiler.java:373)
     at org.apache.jasper.compiler.Compiler.compile(Compiler.java:353)
     at org.apache.jasper.compiler.Compiler.compile(Compiler.java:340)
     at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:646)
     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:357)
     at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
     at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
     at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:7)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.spring.web.JspController.doHandle(JspController.java:13)
     at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:74)
     at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
     at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
     at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
     at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954)
     at org.apache.jsp.tag.web.ext.includeExtension_tag._jspx_meth_c_005fif_005f0(includeExtension_tag.java:122)
     at org.apache.jsp.tag.web.ext.includeExtension_tag.doTag(includeExtension_tag.java:85)
     at org.apache.jsp.tag.web.ext.includeExtensions_tag._jspx_meth_ext_005fincludeExtension_005f0(includeExtensions_tag.java:322)
     at org.apache.jsp.tag.web.ext.includeExtensions_tag._jspx_meth_c_005fforEach_005f2(includeExtensions_tag.java:291)
     at org.apache.jsp.tag.web.ext.includeExtensions_tag.doTag(includeExtensions_tag.java:134)
     at org.apache.jsp.admin.serverConfigGeneral_jsp._jspService(serverConfigGeneral_jsp.java:213)
     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
     at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
     at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
     at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:7)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.spring.web.JspController.doHandle(JspController.java:13)
     at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:74)
     at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
     at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
     at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
     at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:954)
     at org.apache.jsp.tag.web.ext.includeExtension_tag._jspx_meth_c_005fif_005f0(includeExtension_tag.java:122)
     at org.apache.jsp.tag.web.ext.includeExtension_tag.doTag(includeExtension_tag.java:85)
     at org.apache.jsp.admin.adminMain_jsp._jspx_meth_ext_005fincludeExtension_005f0(adminMain_jsp.java:223)
     at org.apache.jsp.admin.adminMain_jsp.access$4(adminMain_jsp.java:212)
     at org.apache.jsp.admin.adminMain_jsp$Helper.invoke3(adminMain_jsp.java:313)
     at org.apache.jsp.admin.adminMain_jsp$Helper.invoke(adminMain_jsp.java:343)
     at org.apache.jsp.tag.web.page_tag.doTag(page_tag.java:670)
     at org.apache.jsp.admin.adminMain_jsp._jspx_meth_bs_005fpage_005f0(adminMain_jsp.java:138)
     at org.apache.jsp.admin.adminMain_jsp._jspService(adminMain_jsp.java:108)
     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
     at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
     at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
     at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:7)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.spring.web.JspController.doHandle(JspController.java:13)
     at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:74)
     at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
     at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
     at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
     at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
     at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
     at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:238)
     at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
     at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1047)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:817)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:6)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at jetbrains.buildServer.web.DiagnosticFilter.runChainWithModifiedThreadName(DiagnosticFilter.java:61)
     at jetbrains.buildServer.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:12)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:4)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
     at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
     at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
     at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
     at java.lang.Thread.run(Thread.java:722)
0
Comment actions Permalink

Also just to tag on but I can run the profiler agent by itself

$ java -agentpath:/opt/TeamCity/bin/libyjpagent.so=help
YourKit Java Profiler 8.0.29 JVMTI agent usage: -agentlib:yjpagent=[help]|[<option>, ...]

Option             Description
------------------ ----------------------------------------
port=<value>       port to listen on (1-65535, by default any free port is chosen)
listen=<ip>:<port> same as 'port', but bind agent socket to particular IP only
onlylocal          allow only local connections to profiled application
dir=<custom dir>   directory where snapshots are created (by default <user home>/Snapshots)
telemetrylimit=<N> limit telemetry buffer capacity to N hours (approximately)
sampling           startup with CPU sampling
tracing            startup with CPU tracing
                   (cannot be used together with 'disablecounts')
noj2ee             do not perform J2EE high level profiling
                   (use only in combination with 'sampling' or 'tracing')
alloceach=<N>      startup with object allocation recording: record each N-th allocation;
                   can be used together with 'allocsizelimit';
                   cannot be used together with 'disablealloc'
allocsizelimit=<B> startup with object allocation recording: record objects with size >= B bytes
                   can be used together with 'alloceach';
                   cannot be used together with 'disablealloc'
monitors           startup with monitor usage profiling
usedmem=<percent>  capture memory snapshot automatically when used heap memory
                   reaches the threshold
onexit=memory      always capture memory snapshot on exit
onexit=snapshot    always capture performance snapshot on exit (i.e. without heap)
disablealloc       do not instrument bytecode for object allocation recording
disablecounts      do not instrument bytecode for CPU tracing
disablej2ee        do not instrument bytecode for J2EE profiling
disablestacktelemetry
                   do not collect thread stack and status information
quiet              suppress diagnostic messages

Please find detailed description in the Help section "Startup options":
http://www.yourkit.com/docs/80/help/additional_agent_options.jsp

Examples:
java -agentlib:yjpagent FooClass
java -agentlib:yjpagent=alloceach=10,allocsizelimit=1000000,dir=c:\MySnapshots FooClass
java -agentlib:yjpagent=usedmem=70 FooClass

0
Comment actions Permalink

Christopher,

Sorry for the inconvenience with the profiling plugin error: it got some code committed compatible only with TeamCity 7.1 EAP. THis is not fixed and you can get a new plugin from http://teamcity.jetbrains.com/viewLog.html?buildId=66995&tab=artifacts&buildTypeId=bt262

Getting profiling snapshot would be perfect, but you can also start with general means like thread dumps, etc.

0
Comment actions Permalink

Thanks Yegor. I have attached the CPU snapshot and the catalina.out with the stack traces. The memory snapshot was too large to attach so let me know if you require it



Attachment(s):
TeamCity-Performance.tar.gz
0
Comment actions Permalink

Christopher,

The snapshot does not reveal any significant load as well as the thread dump.
Was the CPU heavily loaded by TeamCity's java process when you took the snapshot?
Can you doublecheck it's TeamCity server java which is loading the CPU?

If confirmed, please attach CPU load data for the TeamCity java process and take CPU snapshot (start, ensure the process is loading CPU, stop after 5-10 minutes) and attach the data.

WHile most probably not related, I'd recommend to use latest Oracle 1.6 JDK for the server.

0
Comment actions Permalink

Hi Yegor,

I am very sure that it is TeamCity causing the load and its loading the server as it starts rather than needing a trigger to start it off

I have attached an extended CPU profile and put a capture of top from the host running TeamCity


top - 08:20:58 up 271 days, 22:55,  3 users,  load average: 34.04, 33.41, 33.31
Tasks: 158 total,   4 running, 154 sleeping,   0 stopped,   0 zombie
Cpu(s): 34.0%us, 65.5%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.5%si,  0.0%st
Mem:  24738248k total, 20788856k used,  3949392k free,   560692k buffers
Swap: 26836984k total,    30276k used, 26806708k free, 17344476k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
1491 mavsup    20   0 3157m 744m  13m S 253.6  3.1  15562:56 /usr/bin/java -Djava.util.logging.config.file=/opt/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xmx512m -XX:MaxPermSize=270m -Dlog4j.configuration=file:/opt/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djava.awt.headless=true -agentpath:/opt/TeamCity/bin/libyjpagent.so=disab
15048 mavsup    20   0 2722m 264m  14m S 43.7  1.1   3983:52 /usr/java/jdk1.7.0_05/jre/bin/java -ea -Xmx384m -Dteamcity_logs=../logs/ -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml -classpath /opt/TeamCity/buildAgent/lib/agent.jar:/opt/TeamCity/buildAgent/lib/messages.jar:/opt/TeamCity/buildAgent/lib/jdk-searcher.jar:/opt/TeamCity/buildAgent/lib/xml-rpc-wrapper.jar:/opt/TeamCity/buildAgent/lib/commons-collection
14671 mavsup    20   0 7622m  48m 9752 S 22.4  0.2   2629:52 /usr/bin/../bin/java -ea -cp ../launcher/lib/launcher.jar jetbrains.buildServer.agent.Launcher -ea -Xmx384m -Dteamcity_logs=../logs/ -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml jetbrains.buildServer.agent.AgentMain -file ../conf/buildAgent.properties



Attachment(s):
TeamCity-2012-07-19-CPU.snapshot.bz2
0
Comment actions Permalink

Christopher,

Still nothing special in the snapshot.

Reading through your "it looks to have started at midnight on July the 1st for some reason" - it looks very much alike the known leap second Linux kernell issue.
e.g. see http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second

0
Comment actions Permalink

Thanks Yegor, that worked a treat!

0

Please sign in to leave a comment.