TeamCity Projects page takes forever to load.

Hello,

We are having an issue with TeamCity. We are on the latest Release 6.0.3, and our projects pages takes 3-5 minutes to load completely and it continues to get slower. This does not seem to matter which browser we are using as we have tested several.

Our server is on a virtual machine and the setup is:

Windows Server 2008 R2
4 Logical Processor Cores
3072MB Ram
The virtual machine lives on a 5 disk raid 5 array.
10gb backbone in the server room

TeamCity is set to use 1150mb JVM Heap, and 150mb permgen, which is the about the max you can use with the 32 bit tomcat. We currently have 337 active build configs, and we clean up anything older then 10 days.

I've also applied some cache tuning to the MYSQL Database, and as well enabled query cache.

Nothing has seemed to work and the projects page continues to load slower and slower.

Thanks for any help.

12 comments
Comment actions Permalink

Hi Alexander

Colud you take a thread dump please

0
Comment actions Permalink

Michael,

I've attached the thread dump, thank you.

Alex



Attachment(s):
threadDump-2011-05-23_09.35.28.txt.zip
0
Comment actions Permalink

Thanks for the dump.
Unfortunatelly we don't see anything suspicious there.

Could you please open the overview page in a browser, confirm that the server is hanging, and take server dumps at exactly this point of time.
Would be great if you catch 3 dumps with 10-second interval. That would allow us to investigate dynamics of the server performance.

Thanks

0
Comment actions Permalink

Michael,

Any updates? This problem is still plaguing us.

0
Comment actions Permalink

We do not see anything strange in the dumps. Are you sure the problem is in TeamCity itself? Does it eat too much CPU at the moments when all is hanging? Do you have Apache or IIS in front of the TeamCity server, or a proxy in your network?

0
Comment actions Permalink

We are not sure it is server, we are not using apache or IIS. TeamCity is installed as the stand alone version running on a server all by itself. There is no proxy on our network either, we believe it may be related to how the projects page is loaded in the browser.

When you open the projects page, a box pops up and says loading, this happens until it has loaded all the active projects(21 active projects with 349 build configurations). This process can take 3-5 minutes for most of our developers.

0
Comment actions Permalink

What is the size of this page in your case?

0
Comment actions Permalink

It is reporting at 2.78megs.

0
Comment actions Permalink

Let's try to add these parameters to MySQL configuration file (my.ini on Windows):
log-slow-queries
long_query_time=5

You should restart both TeamCity and MySQL after that. MySQL should create slow queries log under its Data directory. Please try to open overview page several times, and attach here queries from this log, if they appear there.

0
Comment actions Permalink

Pavel,

Thanks for your help, I've turned on the slow queries log, and had several users load up the overview page several times and the log remains empty.  I did however notice the wide variation of page load times based on browsers my users were running. I curriously started running my own tests and got these results:

Internet Explorer 8 ~ 20 seconds
Internet Explorer 9 ~ 20 seconds
Chrome best case ~ 30 seconds, worst case 3 minutes
Firefox 3.6.x ~ 5 seconds
Firefox 4.0.x ~ 3 seconds



Chrome appears to be the only/worst offender, and all of my users that were complaining of slow page load times were using chrome, including myself. My thoughts are maybe the way that TeamCity manipulates the DOM after the page has loaded doesn't agree with Chrome very well. For now I think our issue is solved and I will recommend to my users that they no longer use chrome when working with TeamCity.
0
Comment actions Permalink

I would say upgrade to TeamCity 6.5 can help in your case. We reduced size of the page, for example, we no longer load content of collapsed projects. Take a look at this thread: http://devnet.jetbrains.net/message/5305891#5305891

0

Please sign in to leave a comment.