Large slowdown when viewing the "Agent Requirements" section
Hello!
I am running TeamCity v5.0.3 on a Windows Server 2003 32Bit VM. I have created a configuration on that server. With the "Dependencies" section blank, I can switch to the "Agent Requirements" section smoothly and quickly. When I add the large list of dependencies (I count 43 snapshot dependencies), then click on the "Agent Requirements", it takes several minutes to load; regardless of what is in that section (blank or populated). The load process is slow no matter how many times I try to access that section.
On the TeamCity server, I noticed the tomcat6.exe process was pegged at 100% CPU time the entire time it was trying to load the "Agent Requirements" page. I tried restarting the service and also restarting the entire VM. Same result. I tried copying the configuration and the copy exhibited the same slowdown. I then deleted the configuration completely and started a "new" configuration with the same parameters. Slow down appeared but only after I added the dependencies.
I have run this exact same configuration (with the snapshot dependencies included) on a TeamCity v5.0.1 server and this did not happen. Any suggestions? Thanks!
Please sign in to leave a comment.
Could you please take several thread dumps while the agent requirements page is opening?
Read more about thread dumps here:
http://confluence.jetbrains.net/display/TCD5/Reporting+Issues#ReportingIssues-HangsandThreadDumps
Sure can! Here are two thread dumps. I am normally running the TC server as a service, but I ran it from the console to get the thread dump. The issue still occurs (which leads me to believe it's not Tomcat). I started the server, went to edit the configuration, then clicked on "Agent Requirements". After the Agent Requirements loaded (several minutes), I hit CTRL-break and saved the console window info to the .LOG files. I then shut the server down and repeated the process for the second .LOG file.
Attachment(s):
TeamCity5.0.3_DavidW.zip
David, please take thread dumps while the page is loading, not after, otherwise we would not be able to find the cause of the reason.
Oops. Ok, this file, all the thread dumps were taken while the page was loading.
Attachment(s):
TeamCity5.0.3_DavidW.zip
Thank you David, we are investigating this issue.
Thanks for looking at this! I have noticed that it was happening when I would look at "Compatible Configurations" but not as long. I did notice that before I killed the server (which was started via the console), an out of memory java heap error was there. I have since restarted the server from the services window. Don't know if they are related but thought I would pass on the info!
Do the menus I listed require access or communication with the database?
Yes, they are. I think we've found some problems in our code. We applied a couple of patches to TeamCity 5.1, it will work faster on these pages. Not sure we will be able to integrated the patches to 5.0 branch, there were a lot of changes. If you do not plan to upgrade to 5.1 which we hope to release soon, please submit a bug report with your thread dump attached to our tracker. We'll see what we can do in 5.0.
What is the timeframe 5.1 is supposed to be released? If it's soon, I can handle to long delays if it doesn't affect normal usage.
If all will go as planned, 5.1 will be released on the next week.
Very good! I think I can wait until then. I appreciate your help!
One question, just to check the improvement will help in your case. Are there any artifact dependencies configured in build configurations used as snapshot dependencies? What kind of artifact dependencies are used: last pinned, by build number, last finished?
If there are artifact dependencies, and some of them use last pinned builds or builds with specific build number, try to reconfigure them to last finished, just to check whether it helps.
They all use Last Finished Build. But it exhibits the slowdown even before I specify the artifacts and their destination location in the Dependencies section. All I have to do is create the list of SNapshot dependencies (top section, no bottom section). The slowdowns occur if I populate the bottom section of the Dependency section. When it is populated, they are Last Finished Builds.
David, we were able to incorporate a change to 5.0 branch which may help in your case. You can take build with this change from here: ftp://ftp.intellij.net/pub/.teamcity/5.0.4-rc/TeamCity-10845.war
This is a not yet released 5.0.4, it contains bugfixes only. Before installing this war make a backup of your config files and database. To install the war you can replace content of the TEAMCITY_INST_DIR/webapps/ROOT directory with the files from this war (to be able to rollback easily backup the ROOT directory too).
Hope it helps.
Thanks for the 5.0.4 update. Unfortunately, it didn't fix the issues. The menu loads are still very slow.
Could you please attach new thread dumps? Try to take them several times.
Sure can! Here they are. The final dump was taken just after the "Agent Requirements" page loaded.
Attachment(s):
ThreadDump4.zip
Thank you for thread dumps. Today we released TeamCity 5.1 RC, it has a number of speedups that can't be easily integrated in 5.0 builds, but may help in your case (according to your thread dumps). RC build should be relatively stable, but if you choose to upgrade, do not forget to make a backup.
Hello Pavel,
I've noticed that this same configuration, when I try to run it, fails to download all of the artifacts. It comes back with an error "RunBuildException when running build stage DownloadDependenciesStage: Artifacts resolving failed". When I go to "Dependencies" and "Check artifact Dependencies", it is able to find all the artifacts and their paths with no problem. Is this related to the slowdown and will it be fixed in 5.1? Thanks!
Could you please attach build log?
Sure. Is there an email or ftp location I can send it to?
Please use: teamcity-feedback[at]jetbrains.com
Ok sent! Hope that helps out.