TeamCity performance problems
Lately we've encountered performance problems with our TeamCity server (we're currently using 6.0.3 (build 15925)). To the point when you're waiting for the web-page to load for more than 30 minutes. At this moment TeamCity java process is eating up to 100% of CPU, the database service (MySQL) is working fine and not highly loaded, and the log is full of following messages (though I don't know if this is related to the problem):
[2011-06-15 18:27:21,769] WARN - jetbrains.buildServer.AGENT - Registration of agent: build-agent0 {id=0, host=172.16.14.212:9090}@277751c1 failed. Agent with the same name build-agent0 {id=45, host=172.16.14.212:9090}@1c89a3dd is already registered since 09.06.11 11:01
[2011-06-15 18:27:21,797] WARN - jetbrains.buildServer.AGENT - Registration of agent: build-agent15 {id=0, host=172.16.14.3:9090}@4bedfb9d failed. Agent with the same name build-agent15 {id=71, host=172.16.14.3:9090}@2f7f7b33 is already registered since 09.06.11 11:00
[2011-06-15 18:27:21,831] WARN - jetbrains.buildServer.AGENT - Registration of agent: build-agent1 {id=0, host=172.16.10.216:9090}@5492965e failed. Agent with the same name build-agent1 {id=75, host=172.16.10.216:9090}@66c17f01 is already registered since 09.06.11 11:01
[2011-06-15 18:27:29,283] WARN - jetbrains.buildServer.AGENT - Registration of agent: build-agent12 {id=0, host=172.16.10.154:9090}@74d8f1a5 failed. Agent with the same name build-agent12 {id=32, host=172.16.10.154:9090}@18b93be3 is already registered since 14.06.11 18:54
[2011-06-15 18:27:29,286] WARN - jetbrains.buildServer.AGENT - Registration of agent: build-agent16 {id=0, host=172.16.10.10:9090}@5aa16776 failed. Agent with the same name build-agent16 {id=31, host=172.16.10.10:9090}@6b52a96e is already registered since 10.06.11 19:30
[2011-06-15 18:27:29,288] WARN - jetbrains.buildServer.AGENT - Registration of agent: build-agent4 {id=0, host=192.168.194.3:9090}@790921f6 failed. Agent with the same name build-agent4 {id=76, host=192.168.194.3:9090}@391ecf28 is already registered since 09.06.11 11:01
These agents don't lost connection to the server and even can run a build at that moment.
When this happens we have to restart server. But, in about 3-5 hours after the restart the problem appears again.
Appreciate any help.
Please sign in to leave a comment.
Hi
Please take a process thread dump as described at Hangs and Thread Dumps page.
Thanks
Hi,
I'm not sure it is same issue. I see 100% CPU usage by java.exe too.
Mini dump is here: http://dl.dropbox.com/u/62996911/java-mini.7z
Please let me know if you need full dump.
TC 5.1.5
Thanks.
Dima
Hi
Your issue is quite different. Here is workload on agent side.
How much time does the build take in side TeamCity?
Go to \BuildAgent\work directory and run the build script manually from there. How much time does it take in this case? Is CPU also loaded up to 100% ?
Michael
Hi Michael,
Thank you for reply.
The issue is not persistent.I event dont know how to reproduce it. Sometimes Build Agent works good, sometimes java.exe uses 100% of CPU.
As far we have only one CPU on build server, this issue stops build progress.
Anyway, I will check build time tomorrow.
THanks
Dima
TeamCity 7.0 has Performance Monitor, it would help to diagnose such kind of issues.
Hi Michael,
I did not try to run build script maually. I've decided to migrate to 7.0.1. If issue will ocur again I will start investigation.
Thank you for hint.
Best regards
- Dima
Issue has came back. Again java.exe used 100% of CPU.
Going to enable perfmon.
Team City 7.0.1
Attachment(s):
tc-stack.7z.zip
Could you please add more details to your issue.
Where do you experience 100% CPU usage of java. On agent or on server?
Thead dumps you've attached were from agent. Agent was packing files into .zip artchive to publish as artifact.
The load could be caused by a big size of files it was trying to pack.
>> Where do you experience 100% CPU usage of java. On agent or on server?
Issue is on the agent.
>> Agent was packing files into .zip artchive to publish as artifact.
Weird... Agent was building a project. There were no artifacts yet.
Please check artifact paths in the configuration.
There is a service message to publish artifacts from a build:
http://confluence.jetbrains.net/display/TCD7/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-PublishingArtifactswhiletheBuildisStillinProgress
Does your build use it?
Please include agent debug logs as described here:
Reporting Issues - TeamCity 7.x Documentation - Confluence
We use ##teamcity[publishArtifacts '<path>']
I've enabled Debug logging on agent. Let's see if it will help. Meanwhile I've checked current logs and see NullPointerExceptions in the ZIP code. Please find logs attached.
Attachment(s):
exceptions.txt.zip
Hello,
Thank you for logs, I created an issue for it.
http://youtrack.jetbrains.com/issue/TW-21729
Could you please vite for it and add an artifacts pattern and listing of a directory that you want to include into artifact.
Here is link to logs: http://dl.dropbox.com/u/62996911/logs1.7z
How can I provide you with password?
You may email the password to teamcity-feedback@jetbrains.com