Trying to see tests is very very slow

Hello all,

I'm testing TeamCity, and I've integrated some boost "plugin" that
generates output message that TeamCity can handle. It works great, I've
seen all tests results once, but now it's not working anymore..

For example, I have this :

#6 Tests failed: 27 (27 new), passed: 396

If I click to see all tests, it's just waiting, and if I have a look on
server, I see tomcat6 process taking almost 100% of a core, and a java
process under TeamCityAgentService-windows-x86-32.exe taking as well
100% of a core.

And as I was writing this, the Build Agent just crashed, and has been
restarted, still taking same cpu.

Is it normal or not? What can I do? Where to look to see if something is
wrong in my config or smthg else ?

Thanks,

Gérald

22 comments
Comment actions Permalink

No, its not normal. Could you please install this plugin: http://www.jetbrains.net/confluence/display/TW/Server+Profiling
then reproduce this problem with enabled CPU profiling and attach CPU profiling snapshot here or upload it to: ftp://ftp.intellij.net/.uploads

0
Comment actions Permalink

Le 25/05/2009 14:07, Pavel Sher a ecrit :

No, its not normal. Could you please install this plugin: http://www.jetbrains.net/confluence/display/TW/Server+Profiling
then reproduce this problem with enabled CPU profiling and attach CPU profiling snapshot here or upload it to: ftp://ftp.intellij.net/.uploads


I cannot start server with -agentlib:yjpagent parameter...

Doc says :

JVM parameter to the TeamCity server startup script. This can be done by
adding the parameter to the TEAMCITY_SERVER_OPTS environment variable if
you start TeamCity by its scripts.

But what if TeamCity is runned as a service ?

Or how can I run it with scripts ?

Gérald

0
Comment actions Permalink

Le 26/05/2009 10:49, Pavel Sher a ecrit :

Please take a look here: http://www.jetbrains.net/confluence/display/TCD4/TeamCityStartupProperties


Thanks, but it doesn't change anything, when I click on "Start" in
TeamCity Server Profiler, I have a popup saying "To profile application,
you should run it with the profiler agent" I don't understand what it means.

Gérald

0
Comment actions Permalink

Could you please provide a screenshot of the parameters dialog for your TeamCity service (a window opened by tomcat6w.exe)? Did you copy necessary binary files as it is described at the step 3 of the plugin installation procedure (http://www.jetbrains.net/confluence/display/TW/Server+Profiling)?

0
Comment actions Permalink

Le 26/05/2009 11:18, Pavel Sher a ecrit :

Could you please provide a screenshot of the parameters dialog for your TeamCity service (a window opened by tomcat6w.exe)?

Here it is :

Did you copy necessary binary files as it is described at the step 3 of the plugin installation procedure (http://www.jetbrains.net/confluence/display/TW/Server+Profiling)?

I guess not. But actually I don't see what is the "from" in the doc :

copy files from the bin/ directory to the /bin directory. "bin/" ok, but from which folder, I'm not sure documentation is clear enough, or maybe I'm missing something ? Gérald

Le 26/05/2009 11:18, Pavel Sher a ecrit :
)?
I guess not. But actually I don't see what is the "from" in the doc :

copy files from the bin/<your platfom> directory to the <TeamCity Installation Dir>/bin directory.

"bin/<your platfom>" ok, but from which folder, I'm not sure documentation is clear enough, or maybe I'm missing something ?

Gérald

]]>

0
Comment actions Permalink

You should take corresponding files from the bin directory of the plugin files: http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt120&buildId=lastPinned

0
Comment actions Permalink

Le 26/05/2009 12:14, Pavel Sher a ecrit :

You should take corresponding files from the bin directory of the plugin files: http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt120&buildId=lastPinned


I'm desperate... I still have same error message. I put the dll in bin
folder, restarted service, reloaded admin page, but nothing changed.

I join teamcity-server.log, if you can see something that's not normal.

Gérald

INFO - jetbrains.buildServer.SERVER - Starting TeamCity server
INFO - dServer.serverSide.ServerPaths - TeamCity data path was read from teamcity.data.path
INFO - dServer.serverSide.ServerPaths - TeamCity data directory:d:/teamcity/.BuildServer
INFO - rverSide.impl.DatabaseSettings - External database settings not specified: cannot find D:\TeamCity\.BuildServer\config\database.properties
INFO - erver.serverSide.impl.DBFacade - Connection pool configured (org.hsqldb.jdbcDriver, maximum number of connections: 50)
INFO - erver.serverSide.impl.DBFacade - Connection pool URL: jdbc:hsqldb:file:D:/TeamCity/.BuildServer/system/buildserver
INFO - jetbrains.buildServer.SERVER - Loading roles from the configuration file: D:\TeamCity\.BuildServer\config\roles-config.xml
INFO - jetbrains.buildServer.SERVER - Start initializing build durations cache...
INFO - jetbrains.buildServer.SERVER - Finished initializing build durations cache
INFO - n.email.EMailNotificatorConfig - Loading configuration file: D:\TeamCity\.BuildServer\config\email-config.xml
INFO - ion.template.NotificatorConfig - Loading configuration file: D:\TeamCity\.BuildServer\config\ide-notificator-config.xml
INFO - jabber.JabberNotificatorConfig - Loading configuration file: D:\TeamCity\.BuildServer\config\jabber-config.xml
INFO - ion.template.NotificatorConfig - Loading configuration file: D:\TeamCity\.BuildServer\config\win32-config.xml
INFO - rver.plugins.PluginManagerImpl - ===========================================================
INFO - rver.plugins.PluginManagerImpl - Scanning plugin folder: D:\TeamCity\webapps\ROOT\WEB-INF\plugins
INFO - rver.plugins.PluginsCollection - Found plugin buildDurationStatisticsPlugin
INFO - rver.plugins.PluginsCollection - Found plugin changeViewers
INFO - rver.plugins.PluginsCollection - Found plugin clearcase
INFO - rver.plugins.PluginsCollection - Found plugin cvs
INFO - rver.plugins.PluginsCollection - Found plugin feed
INFO - rver.plugins.PluginsCollection - Found plugin ldap
INFO - rver.plugins.PluginsCollection - Found plugin perforce
INFO - rver.plugins.PluginsCollection - Found plugin remoteAccess
INFO - rver.plugins.PluginsCollection - Found plugin stacktraces
INFO - rver.plugins.PluginsCollection - Found plugin starteam
INFO - rver.plugins.PluginsCollection - Found plugin svn
INFO - rver.plugins.PluginsCollection - Found plugin TeamCityArtifactsSizeStatisticsPlugin
INFO - rver.plugins.PluginsCollection - Found plugin tfs
INFO - rver.plugins.PluginsCollection - Found plugin vss
INFO - rver.plugins.PluginsCollection - Found plugin FxCop
INFO - rver.plugins.PluginsCollection - Found plugin rake-runner
INFO - rver.plugins.PluginsCollection - Found plugin xml-report-plugin
INFO - rver.plugins.PluginManagerImpl - Scanning plugin folder: D:\TeamCity\.BuildServer\plugins
INFO - rver.plugins.PluginsCollection - Found plugin serverProfile
INFO - rver.plugins.PluginManagerImpl - Found 18 plugins:
INFO - rver.plugins.PluginManagerImpl - Start intializing plugins...
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\cvs\agent\cvsAgent.zip for cvs
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\remoteAccess\agent\remoteAccess.zip for remoteAccess
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\stacktraces\agent\stacktracesAgent.zip for stacktraces
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\svn\agent\svnAgent.zip for svn
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\tfs\agent\tfs-agent.zip for tfs
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\.unpacked\fxcop-plugin\agent\fxcop-agent.zip for FxCop
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\.unpacked\rake-runner-plugin\agent\rake-runner-agent.zip for rake-runner
INFO - ns.agent.AgentPluginsPublisher - Registering agent plugin D:\TeamCity\webapps\ROOT\WEB-INF\plugins\.unpacked\xml-report-plugin\agent\xml-report-plugin.zip for xml-report-plugin
INFO - rver.plugins.PluginsCollection - No plugins were loaded with standalone classloaders
INFO - rver.plugins.PluginsCollection - Load shared classloader for 18 plugins
INFO - jetbrains.buildServer.LDAP - -


LdapManager Start -



INFO - rver.plugins.PluginManagerImpl - Loaded 18 plugins:
INFO - rver.plugins.PluginManagerImpl - ===========================================================
INFO - jetbrains.buildServer.CLEANUP - DB compact time has been set to 03:00:00 daily
INFO - jetbrains.buildServer.SERVER - Loading history of VCS changes...
INFO - jetbrains.buildServer.SERVER - Done loading VCS history: 86 changes loaded
INFO - jetbrains.buildServer.SERVER - Server configuration reloaded.
INFO - jetbrains.buildServer.SERVER - Start prefetching changes for 0 build configurations...
INFO - jetbrains.buildServer.SERVER - Done prefetching changes
INFO - jetbrains.buildServer.SERVER - Start loading cache of build committers...
INFO - jetbrains.buildServer.SERVER - Done loading cache of build committers
INFO - jetbrains.buildServer.SERVER - TeamCity 4.5.1 (build 8975) initialized, OS: Windows XP, JRE: 1.5.0_12-b04
INFO - jetbrains.buildServer.SERVER - TeamCity is running in professional mode
INFO - verSide.impl.BuildServerRunner - Start daemons
INFO - ildServer.IprSettingsPreloader - ==> Start preloading IPR settings for build configurations
INFO - ildServer.IprSettingsPreloader - ==> Done preloading IPR settings for build configurations
INFO - e.impl.agent.PluginsLookupTask - Build agent distribution has changed on disk. Initiate upgrade.
WARN - jetbrains.buildServer.SERVER - To profile application, you should run it with the profiler agent
java.lang.Exception: To profile application, you should run it with the profiler agent
at com.yourkit.api.Controller.getUsedPort(Controller.java:120)
at com.yourkit.api.Controller.]]>(Controller.java:60)
at jetbrains.buildServer.profile.Profiler.startProfiler(Profiler.java:54)
at jetbrains.buildServer.profile.ProfilerController.doAction(ProfilerController.java:70)
at jetbrains.buildServer.profile.ProfilerController.access$000(ProfilerController.java:20)
at jetbrains.buildServer.profile.ProfilerController$1.handleRequest(ProfilerController.java:42)
at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:29)
at jetbrains.buildServer.profile.ProfilerController.doHandle(ProfilerController.java:39)
at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:60)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:523)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:463)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at jetbrains.spring.web.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:18)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:7)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
WARN - jetbrains.buildServer.SERVER - To profile application, you should run it with the profiler agent
java.lang.Exception: To profile application, you should run it with the profiler agent
at com.yourkit.api.Controller.getUsedPort(Controller.java:120)
at com.yourkit.api.Controller.]]>(Controller.java:60)
at jetbrains.buildServer.profile.Profiler.startProfiler(Profiler.java:54)
at jetbrains.buildServer.profile.ProfilerController.doAction(ProfilerController.java:70)
at jetbrains.buildServer.profile.ProfilerController.access$000(ProfilerController.java:20)
at jetbrains.buildServer.profile.ProfilerController$1.handleRequest(ProfilerController.java:42)
at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:29)
at jetbrains.buildServer.profile.ProfilerController.doHandle(ProfilerController.java:39)
at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:60)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:523)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:463)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at jetbrains.spring.web.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:18)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:7)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
WARN - jetbrains.buildServer.SERVER - To profile application, you should run it with the profiler agent
java.lang.Exception: To profile application, you should run it with the profiler agent
at com.yourkit.api.Controller.getUsedPort(Controller.java:120)
at com.yourkit.api.Controller.]]>(Controller.java:60)
at jetbrains.buildServer.profile.Profiler.startProfiler(Profiler.java:54)
at jetbrains.buildServer.profile.ProfilerController.doAction(ProfilerController.java:70)
at jetbrains.buildServer.profile.ProfilerController.access$000(ProfilerController.java:20)
at jetbrains.buildServer.profile.ProfilerController$1.handleRequest(ProfilerController.java:42)
at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:29)
at jetbrains.buildServer.profile.ProfilerController.doHandle(ProfilerController.java:39)
at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:60)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:523)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:463)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at jetbrains.spring.web.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:18)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:7)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)

0
Comment actions Permalink

Please try to attach screenshot of TeamCity parameters again, previous one is not showing for some reason.

0
Comment actions Permalink

Le 26/05/2009 14:14, Gérald Fauvelle a ecrit :

Le 26/05/2009 12:14, Pavel Sher a ecrit :

>> You should take corresponding files from the bin directory of the plugin files: http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt120&buildId=lastPinned


I'm desperate... I still have same error message. I put the dll in bin
folder, restarted service, reloaded admin page, but nothing changed.


I tried running server in command line, it seems to be better at the
beginning :

Using JVMTI (1.5.0_12-b04;Sun
Microsystems Inc.;mixed mode, sharing;Windows;32 bit JVM)
Profiler agent is listening on port 10001...
*** HINT ***: To get profiling results,
connect to the application from the profiler UI

But server looks for data directory at the wrong place :

TeamCity data directory: C:/Documents and Settings/fauvelle/.BuildServer

instead of d:\TeamCity\.BuildServer, how can I give it the right directory ?

0
Comment actions Permalink

Le 26/05/2009 14:47, Pavel Sher a ecrit :

Please try to attach screenshot of TeamCity parameters again, previous one is not showing for some reason.


Hope it's good this time.

Gérald

0
Comment actions Permalink

No, I still do not see any screenshots. Do you post to the forum from a news client? Please try to create message from the web browser.

0
Comment actions Permalink

Yes, it was from news client. Now it's from web browser.



Attachment(s):
screenshot.png
0
Comment actions Permalink

JVM parameter should be: -agentlib:yjpagent without -D prefix.

0
Comment actions Permalink

I thought I tried that at first and it was crashing... Well, now it runs, but it's same problem as with command line, it doesn't look for teamcity data folder at right place...

0
Comment actions Permalink

Le 26/05/2009 16:11, Gérald Fauvelle a ecrit :

I thought I tried that at first and it was crashing... Well, now it runs, but it's same problem as with command line, it doesn't look for teamcity data folder at right place...


Ok, sorry for that, it's in the documentation...
Adding -Dteamcity.data.path=D:\TeamCity\.BuildServer solves the problem.

And now profiling works !

Thank you for your help.

Gérald

0
Comment actions Permalink

There must be additional parameter -Dteamcity.data.path=..., according to your screenshot it was specified before, please check it is still there.

0
Comment actions Permalink

Ok, I have uploaded snapshot, name is TeamCity-2009-05-26.snapshot.

I clicked on "Test failed... " link in one of my build, and waited about 2 minutes, then capture a snapshot.

Tell me if you can find something or if you need something else.

0
Comment actions Permalink

It seems most of the time is spent on reading the test output. What is the average size of the output + stacktrace of a failed test in the build which tests you wanted to see?

0
Comment actions Permalink

Le 26/05/2009 17:31, Pavel Sher a ecrit :

It seems most of the time is spent on reading the test output. What is the average size of the output + stacktrace of a failed test in the build which tests you wanted to see?

>

It shouldn't be that big, but I think that I know what wrong.
Before using TeamCity I was dumping that in xml output, that was small
and quick.

With TeamCity, I used the boost plugin that activates all logs,
including all BOOST_CHECK that are done, so that must have increase
output datas in very big proportions.

I'll check that to see if I can reduce output.

In case of an external database for TeamCity, are there datas stored in
database as well ?


Gérald

0
Comment actions Permalink

No, build logs are stored on disk under the .BuildServer/system/messages directory.

0
Comment actions Permalink

Le 26/05/2009 17:31, Pavel Sher a ecrit :

It seems most of the time is spent on reading the test output. What is the average size of the output + stacktrace of a failed test in the build which tests you wanted to see?


That was too much log of boost test. Logging only the failed checkers is
much better, now displaying test is very quick.

Thank you for your help

Gérald

0

Please sign in to leave a comment.