Problem with agent / server communication

Are there any errors in the teamcity-server.log on the server side?

Hi Pavel,

Thanks for the reply.  There are no errors on the server side.  There are no log messages at all anytime during the duration of the build in question.  Also, to give you a little information about our environment.  We have 232 build configurations across 11 projects with one server and 19 agents all running on Windows XP.

Will

Please install this plugin: http://www.jetbrains.net/confluence/display/TW/Server+Profiling

Then you can enable CPU profiling and reload build page several times. Upload CPU snapshot to our ftp: ftp://ftp.intellij.net/.uploads

BTW have you upgraded to TeamCity 4.0.1?

We are using 4.0.1.

I'm having a problem getting the profiling plugin to work.  I copied the 2 jar files to the WEB-INF/lib directory, copied the 2 files to the /bin directory and added the -agentlib:yjpagent parameter to the Java Options of the Tomcat startup.  I'm running it as a service, so I added the option to the Java tab in the configuration dialog.  However, I'm not seeing the profiling option in the Sever Administration page.

Profiling options will be shown on the Server configuration page of the Administration area. The slowdown you experiencing can also be caused by search plugin indexing tests in your builds. At the same page (Server configuration) you can see status of the indexing process. Please check how much builds are queued for indexing. Try to disable indexing of tests and see whether it helps. To do so start TeamCity with this additional parameter:
-Dtc.search.index.tests=false

Currently there are no builds queued for indexing:

Search

I've enabled CPU profiling as you requested and uploaded the file to your FTP server.

Thanks, Will

How many running builds do you have at the moment? Are there builds running simultaneously in the one build configuration and having many tests failing? Please also describe your hardware, and memory settings specified for TeamCity.

Also what is the number of tests in build configuration you mentioned in the first message.

Hi Pavel,

We have anywhere from 1 to 15 concurrent running builds at any given time.
There is at most one running build for this particular build configuration running concurrently.
There are currently around 85 tests failing in this build.
Our TeamCity server and most of our agents share the same configuration:

• Windows 32-bit VM
• ~2.5 GHz single CPU
• 2 GB of Memory

These VMs are dedicated to TeamCity and are not being used for anything else.  The server is currently allocated 1.25GB max memory and seems to use around 700MB or so usually.
There are 818 tests currently in the build in question.

Thanks, Will

Please try to install the latest version of the profiler plugin and capture snapshot with J2EE profiling option enabled.

Hi Pavel,

I've installed the latest Jars from this location: http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt120&buildId=lastPinned, but I don't see an option to enable J2EE profiling.  Is there a special step to do that?

Will

Are you sure you removed previous jar files? Could you please check that jar files in the WEB-INF/lib correspond to new plugin jars? Try to completely reinstall plugin.

Here are the steps I performed:

1. Stopped the server
2. deleted the 2 jar files from the WEB-INF/lib directory
3. deleted the dll from the bin directory
4. Copied the 2 new jar files to the WEB-INF/lib directory
5. copied the new dll to the bin directory
6. Cleared the Catalina work directory
7. Started the server

I still do not see a J2EE profiling option in the server configuration screen.

Thanks, Will

Ok, let's do the following:
- create folder .BuildServer/plugins/serverProfile
- move jar files to this folder
- remove jar files from the WEB-INF/lib
- start server again

What you suggested worked and I was able to enable J2EE Profiling.  I've uploaded a snapshot to your server where I clicked on the Build Overview 3 times.  The first time was for a build with ~70 failed tests, the second time was for a successful build, and the third time was for a build with ~40 failed tests.

Please take a look at this thread: http://www.jetbrains.net/devnet/message/5230263#5230263
Try to install the build mentioned in the last message. Do not forget to backup your data (database and .BuildServer) before installing.

Hello Will,

  We've made another fix which may help to improve performance in your case. Please check the following build:
   ftp://ftp.intellij.net/.1283517263/TeamCity-8198.war (it includes previous enhancement as well)

  Regards,
  KIR

Hi Kirill,

What is the proper procedure for installing the Beta WAR build?  Clearing out everything under webapps\ROOT and dropping in the new WAR?

Thanks for your help, Will

Hi Will,

  cd TeamCity/webapps/
  mv ROOT /some/backup/dir/outside
  mkdir ROOT
  cd ROOT
  jar xf /downloaded/TeamCity.war

  Something like that.

  KIR

Do not forget to copy Oracle JDBC driver to WEB-INF/lib

Thanks Pavel and Kirill,

I will install the Beta WAR this weekend and let you know how it goes.  We need to minimize disruptions to the TeamCity environment during normal working hours as we have close to 200 developers relying on TeamCity for constant feedback.

Thanks again, Will

Have you got any chance to install the build with performance improvement?

Hi Pavel,

I did get a chance to install build 8198 yesterday morning.  It seems a little more responsive, but it still can take a minute or more to load the build overview page for a build with 50 or more failed tests.

Will

Since then we've made more fixes. The last build which contains more performance fixes is here: ftp://ftp.intellij.net/.1283517263/TeamCity-8204.war