Is there any way to speed up the loading of large build log files in the web interface?

Hello Marina,

I have an example of a log file that is only 862KB.  Anyway, this log file slows down the views so much that when I click to see the build details, it takes about a minute and a half to open the details page (this is a failed build, so I assume it is parsing the log file to show the tail of where the error occurred.  Viewing the details page for a successful build opens much snappier).  Then, I clicked the build log tab and it took a full 2 minutes before the view changed to the build log.  When I click to download the full build log, it took about 2 minutes before the browser received a response that a file needed to be saved.  Switching from Important Messages to All Messages took about a minute.  Anyway, this is a smaller log file example that has content from a failed build.  A successful build in our case has a log file about 3 MB in size and clicking the build log tab takes a good 8 - 10 minutes to load.  For that size of log file, it takes 1.5 minutes to reveive the response to download the full build log and switching from Important Messages to All Messages takes x minutes.  These times are consistent for all users across most browsers (Actually, our IE6 users report even worse performance when dealing with log files) and it is affecting us from effectively using the system, because it is difficult to see why a build configuration fail because it takes so long to get access to our log files.

Now one thing that we do have that I should probably mention is that our file data is stored on a RAID10 NAS and the master server accesses content from the NAS over a Gigabit link.  We did the setup this way because the artifacts and everything TeamCity related is critical data that we need to have on fault tolerant and redundant storage.

Thanks for the quick response, and I look forward to hearing your opinion about this performance issue.

Regards

Hamad Deshmukh

TeamCity does process the build log on rendering/download, but usually this  should not be an issue and 2 minutes delay on 1Mb file is certainly not the way it  should be.

We will need more information so that we can investigate the  root cause.

How NAS performace differs from the local disk? Is it  possible to isolate NAS-related issues e.g. by installing a test TeamCity  installation using a local disk?

Do you experience delays only on the  build log-related pages?
Do you have the whole .BuildServer located on NAS?  If yes, do you experience alike issues when downloading artifacts form the  TeamCity UI?

What is the CPU load on TeamCity server usually and when you  see the delays on build log file downloading?

Is it possible to send us build log raw files (located at  .BuildServer/system/messges/<build_id>.*) ?

This problem is pretty strange. Here at Jetbrains sometimes we have logs of size > 100Mb and opening them works fine (if browser has enough memory). I would suggest to install this plugin: http://www.jetbrains.net/confluence/display/TW/Server+Profiling

Read more about profiling here: http://www.jetbrains.net/confluence/display/TCD5/Reporting+Issues#ReportingIssues-ServerPerformance
On the same page you will find information on how to send CPU snapshot to us.

Thanks for the responses Pavel and Marina

Here are some answers to your questions Marina

How NAS performace differs from the local disk? Is it  possible to isolate NAS-related issues e.g. by installing a test TeamCity  installation using a local disk?
Answer: We could try to do this, but in general we don't seem to be having many issues with file access performance on the NAS

Do you experience delays only on the  build log-related pages?
Answer: Yes

Do you have the whole .BuildServer located on NAS?  If yes, do you experience alike issues when downloading artifacts form the  TeamCity UI?
Answer:  Yes the whole .BuildServer is located on the NAS.  But we do not experience the same issues downloading artifacts from TeamCity UI (except for maybe the download all artifacts as a zip link which does seem to take a bit longer than directly downloading artifact files alone)

What is the CPU load on TeamCity server usually and when you  see the delays on build log file downloading?
Answer: Minimal CPU Load.  The teamcity master is set up as a master only (no agent on it) and the only thing installed on it is the teamcity master server and the mysql server that serves as the backend to teamcity.

Is it possible to send us build log raw files (located at  .BuildServer/system/messges/<build_id>.*) ?
Answer: I will look into sending you the raw log files as soon as I turn profiling on as per Pavel's suggestion

Thanks so much for your continued support.  I will send you more data asap.

Regards

Pavel and Marina,

I have just finished configuring the profiling plugin and enabled debug logging to help zero in on the problem.  Sorry it took a bit of time...I was busy with a deadline.  Anyways, I plan to have some information for you in a couple of days once I get the chance to restart the server overnight.

Thanks