Performance and locks on loading build results

We seem to be intermittently getting a problem where TeamCity has a huge delay (5 minutes plus) in opening the build results to look at the logs.  Sometimes it appears to be that a log is completely locked, and other times there is a very long delay.

There is not a high CPU use on the server during this time, and we often revert to running a new build and looking at the new log rather than waiting for the previous one to load (as I can't see any way of viewing the logs beyone the web interface).  We are using Teamcity 6.0.2.

Has anyone seen anything similar, or have any suggestions?

Thanks,
David

7 comments
Comment actions Permalink

Hi David

To investigate this issue we would need additional info

  1. Server thread dump. Please take it at the point of time when the server is hanging.
  2. Build log in a binary format. Open a build page, find internal build ID in URL (like http://localhost:8082/viewLog.html?buildTypeId=bt56&buildId=635). Go to TeamCity data directory, and find corresponding msg5-file (like c:\TeamCity\.BuildServer\system\messages\CH35\635.msg5)

Thanks
Michael
0
Comment actions Permalink

Hi Michael,

Good timing as the loading the build results is freezing again now!

I have attached the files you asked for - if you need anything else please let me know.

Thanks in advance,

David



Attachment(s):
5215.msg5.i1.zip
5215.msg5.zip
threadDump-2011-05-23_18.03.49.txt.zip
0
Comment actions Permalink

Hello, David,

The thread dump shows that TFS issue tracker integration activity occurs to slow down the Build results page rendering.
You can try uninstalling tfsissuetracker.zip and check if performance comes to normal.
I've also passed the issue to TFS issue tracker integration plugin author.

0
Comment actions Permalink

While the TFS issue tracker integration plugin can probably be improved to provide better performance, I also filed an issue to reduce the mpact of a slow plugin on web UI for this case: TW-16985.

0
Comment actions Permalink

Hello David,

I am the author of the plugin. As Victory said it's probably the cause of the slowdown, let me explain briefly.

TeamCity reasons about issues in a very specific way, expecting the ID of the issue to be embedded in the commit comment, and then passing the ID of the issue to the issue tracker integration plugin in a very straightforward way, which then returns the details of the issue to the Web UI.
TFS does not by default work this way and in order to support this scenario in the past I asked for a change in the API to allow supporting TFS by providing more information, specifically the ID of the changeset.

Using the changeset ID I could use the TFS API to lookup the Work Items associated to the changeset, therefore adding an additional step in the workflow TC is expecting. In the common case parsing the ID of the issue from the commit comment is a very fast operation of course, and TC expects it to be fast, but with TFS this is not fast as it entails querying TFS to obtain them. Once the first call returns the list of the Work Items IDs (called IssueMentions in the TC API), TC asks the plugin to retrieve detais about each issue, thus going back to TFS once again.

Both operations are cached thanks to the default implementation of the issue tracker integration plugin, but nevertheless I am confident that is exactly the cause of the slowness you are experiencing because the first operation, which as I said TC expects to be extremely fast, is istead quite a slow operation in TFS.

0
Comment actions Permalink

After your post, I was looking around at the options in the third party plugin, and relaised it was connecting as the account of someone that had left the company at around the time the performance issues started.  I was not aware that the issue tracker used a different account to connect to TFS to the rest of teamcity.

Anyway - I suspect this was the cause (embarassingly), I guess the plugin does not report to say that the user could not be authenticated?

I have updated the username/password and it now seems to be behaving itself.In any case, I will keep an eye on it in any case and fingers crossed...

0
Comment actions Permalink

By "the rest of teamcity" I guess you mean version control, right? They are two independent plugins therefore they don't share the user account. At the moment the plugin does not report explicitly about authentication issues I'm afraid.

0

Please sign in to leave a comment.