Problem with agent / server communication

Ever since the upgrade from TeamCity 3 --> 4, we've been having a variety of different issues.  The issue I'm having currently is with a particular build configuration.  This is a Maven2 build running tests with surefire/junit4.  Everything runs fine until the integration-test phase of the final module in our multi-module build.  I can see by looking in the surefire output directory that the tests are running quickly and the agent log indicates the surefire files are being parsed, however the log messages are not making it to the server.  I'm seeing errors in the log like:


[2009-01-09 14:59:33,937]  DEBUG -    jetbrains.buildServer.AGENT - Agent ping from server
[2009-01-09 14:59:39,390]  DEBUG - .httpclient.HttpMethodDirector - Closing the connection.
[2009-01-09 14:59:39,390]  DEBUG - mons.httpclient.HttpConnection - enter HttpConnection.close()
[2009-01-09 14:59:39,390]  DEBUG - mons.httpclient.HttpConnection - enter HttpConnection.closeSockedAndStreams()
[2009-01-09 14:59:39,390]  DEBUG - .httpclient.HttpMethodDirector - Method retry handler returned false. Automatic recovery will not be attempted
[2009-01-09 14:59:39,390]  DEBUG - mons.httpclient.HttpConnection - enter HttpConnection.releaseConnection()
[2009-01-09 14:59:39,390]  DEBUG - mons.httpclient.HttpConnection - Releasing connection back to connection manager.
[2009-01-09 14:59:39,390]  DEBUG - mons.httpclient.HttpConnection - enter HttpConnection.releaseConnection()
[2009-01-09 14:59:39,390]  DEBUG - mons.httpclient.HttpConnection - Releasing connection back to connection manager.
[2009-01-09 14:59:39,390]  DEBUG - r.agent.impl.AgentLogProxyImpl - Send 101 log messages FAIL
[2009-01-09 14:59:39,390]  DEBUG - r.agent.impl.AgentLogProxyImpl - Send 101 log messages FAIL



 

The behavior is happening on multiple agents.  Also, not sure if this is related, but build overview page is very slow to respond and sometimes takes several minutes to render.  Other pages, like the Project view respond quickly.

25 comments
Comment actions Permalink

Are there any errors in the teamcity-server.log on the server side?

0
Comment actions Permalink

Hi Pavel,

Thanks for the reply.  There are no errors on the server side.  There are no log messages at all anytime during the duration of the build in question.  Also, to give you a little information about our environment.  We have 232 build configurations across 11 projects with one server and 19 agents all running on Windows XP.

Will

0
Comment actions Permalink

Please install this plugin: http://www.jetbrains.net/confluence/display/TW/Server+Profiling

Then you can enable CPU profiling and reload build page several times. Upload CPU snapshot to our ftp: ftp://ftp.intellij.net/.uploads

0
Comment actions Permalink

BTW have you upgraded to TeamCity 4.0.1?

0
Comment actions Permalink

We are using 4.0.1.

I'm having a problem getting the profiling plugin to work.  I copied the 2 jar files to the WEB-INF/lib directory, copied the 2 files to the /bin directory and added the -agentlib:yjpagent parameter to the Java Options of the Tomcat startup.  I'm running it as a service, so I added the option to the Java tab in the configuration dialog.  However, I'm not seeing the profiling option in the Sever Administration page.

0
Comment actions Permalink

Profiling options will be shown on the Server configuration page of the Administration area. The slowdown you experiencing can also be caused by search plugin indexing tests in your builds. At the same page (Server configuration) you can see status of the indexing process. Please check how much builds are queued for indexing. Try to disable indexing of tests and see whether it helps. To do so start TeamCity with this additional parameter:
-Dtc.search.index.tests=false

0
Comment actions Permalink

Currently there are no builds queued for indexing:

Search

Index Status
Version tag: version.23
Disk size: 135.08Mb
Service Status
Builds Indexed: 5914
Builds Queued: 0


 


I've enabled CPU profiling as you requested and uploaded the file to your FTP server.

Thanks, Will

0
Comment actions Permalink

How many running builds do you have at the moment? Are there builds running simultaneously in the one build configuration and having many tests failing? Please also describe your hardware, and memory settings specified for TeamCity.

0
Comment actions Permalink

Also what is the number of tests in build configuration you mentioned in the first message.

0
Comment actions Permalink

Hi Pavel,

We have anywhere from 1 to 15 concurrent running builds at any given time.
There is at most one running build for this particular build configuration running concurrently.
There are currently around 85 tests failing in this build.
Our TeamCity server and most of our agents share the same configuration:

  • Windows 32-bit VM
  • ~2.5 GHz single CPU
  • 2 GB of Memory


These VMs are dedicated to TeamCity and are not being used for anything else.  The server is currently allocated 1.25GB max memory and seems to use around 700MB or so usually.
There are 818 tests currently in the build in question.

Thanks, Will

0
Comment actions Permalink

Please try to install the latest version of the profiler plugin and capture snapshot with J2EE profiling option enabled.

0
Comment actions Permalink

Hi Pavel,

I've installed the latest Jars from this location: http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt120&buildId=lastPinned, but I don't see an option to enable J2EE profiling.  Is there a special step to do that?

Will

0
Comment actions Permalink

Are you sure you removed previous jar files? Could you please check that jar files in the WEB-INF/lib correspond to new plugin jars? Try to completely reinstall plugin.

0
Comment actions Permalink

Here are the steps I performed:

  1. Stopped the server
  2. deleted the 2 jar files from the WEB-INF/lib directory
  3. deleted the dll from the bin directory
  4. Copied the 2 new jar files to the WEB-INF/lib directory
  5. copied the new dll to the bin directory
  6. Cleared the Catalina work directory
  7. Started the server


I still do not see a J2EE profiling option in the server configuration screen.

Thanks, Will

0
Comment actions Permalink

Ok, let's do the following:
- create folder .BuildServer/plugins/serverProfile
- move jar files to this folder
- remove jar files from the WEB-INF/lib
- start server again

0
Comment actions Permalink

What you suggested worked and I was able to enable J2EE Profiling.  I've uploaded a snapshot to your server where I clicked on the Build Overview 3 times.  The first time was for a build with ~70 failed tests, the second time was for a successful build, and the third time was for a build with ~40 failed tests.

0
Comment actions Permalink

Please take a look at this thread: http://www.jetbrains.net/devnet/message/5230263#5230263
Try to install the build mentioned in the last message. Do not forget to backup your data (database and .BuildServer) before installing.

0
Comment actions Permalink

Hello Will,

  We've made another fix which may help to improve performance in your case. Please check the following build:
   ftp://ftp.intellij.net/.1283517263/TeamCity-8198.war (it includes previous enhancement as well)

  Regards,
  KIR

0
Comment actions Permalink

Hi Kirill,

What is the proper procedure for installing the Beta WAR build?  Clearing out everything under webapps\ROOT and dropping in the new WAR?

Thanks for your help, Will

0
Comment actions Permalink

Hi Will,

  cd TeamCity/webapps/
  mv ROOT /some/backup/dir/outside
  mkdir ROOT
  cd ROOT
  jar xf /downloaded/TeamCity.war

  Something like that.

  KIR

0
Comment actions Permalink

Do not forget to copy Oracle JDBC driver to WEB-INF/lib

0
Comment actions Permalink

Thanks Pavel and Kirill,

I will install the Beta WAR this weekend and let you know how it goes.  We need to minimize disruptions to the TeamCity environment during normal working hours as we have close to 200 developers relying on TeamCity for constant feedback.

Thanks again, Will

0
Comment actions Permalink

Have you got any chance to install the build with performance improvement?

0
Comment actions Permalink

Hi Pavel,

I did get a chance to install build 8198 yesterday morning.  It seems a little more responsive, but it still can take a minute or more to load the build overview page for a build with 50 or more failed tests.

Will

0
Comment actions Permalink

Since then we've made more fixes. The last build which contains more performance fixes is here: ftp://ftp.intellij.net/.1283517263/TeamCity-8204.war

0

Please sign in to leave a comment.