Mac OS X agents: hang up
Hello Jetbrains and TC community!
Our company has issues for Mac OS X. They are hanging up periodicaly in different phase of build proccess.
If I press "Stop" for the hanging build - nothing happens.
If I make a limitation for successful build - it will not be failed after time limit.
We don't know, what to do and how to fix or debug it.
If we don't watch the agents page on TC server - we have builds hanging up for 30 hours and more.
Here is some (poor) information:
1) "View thread dump" on TC server:
TeamCity was unable to locate any processes of this build.
2) TC agent logs:
Paulie-Pennino:logs qa$ ls -asl | grep "2 Jul 15:"
0 drwxr-xr-x 18 qa staff 612 2 Jul 15:18 .
10112 -rw-r--r-- 1 qa staff 5176851 2 Jul 15:21 output.log
5848 -rw-r--r-- 1 qa staff 2990096 2 Jul 15:25 teamcity-agent.log
16688 -rw-r--r-- 1 qa staff 8543728 2 Jul 15:21 teamcity-build.log
20488 -rw-r--r-- 1 qa staff 10485828 2 Jul 15:18 teamcity-build.log.1
2880 -rw-r--r-- 1 qa staff 1474473 2 Jul 15:05 teamcity-vcs.log
Paulie-Pennino:logs qa$ tail -10 teamcity-agent.log
[2014-07-02 15:21:28,407] INFO - jetbrains.buildServer.AGENT - Updating agent parameters on the server: AgentDetails{Name='Paulie Pennino', AgentId=42, BuildId=190692, AgentOwnAddress='null', AlternativeAddresses=[192.168.21.94], Port=9090, Version='27767', PluginsVersion='27767-md5-57a86f4200ced1f3112dc533439269b7', AvailableRunners=[Ant, Duplicator, gradle-runner, Inspection, Ipr, JPS, Maven2, rake-runner, simpleRunner, Xcode], AvailableVcs=[cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='c89b1691a521b700cecac0a356e29100'}
[2014-07-02 15:25:41,179] INFO - s.agent.StacktracesAgentPlugin - Start getting thread dumps
[2014-07-02 15:25:41,179] INFO - rocesses.ProcessTreeTerminator - Using jetbrains.buildServer.processes.ProcessTreeTerminatorMac
[2014-07-02 15:25:41,227] INFO - stacktraces.agent.DoThreadDump - Running processes:
[2014-07-02 15:25:41,227] INFO - acktraces.agent.nix.JdkLocator - Looking for JPS in agent-running jvm...
[2014-07-02 15:25:41,232] INFO - acktraces.agent.nix.JdkLocator - Lookup for JPS. Found: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/jps
[2014-07-02 15:25:41,232] INFO - r.stacktraces.agent.nix.JpsRun - Fetcing list of running java processes
[2014-07-02 15:25:41,232] INFO - r.stacktraces.agent.nix.JpsRun - Starting jps: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/jps -q
[2014-07-02 15:25:42,016] INFO - r.stacktraces.agent.nix.JpsRun - Found java processes: [66673]
[2014-07-02 15:25:42,016] INFO - aces.agent.nix.NixDoThreadDump - detected java processes: [66673]
3) TC build log (as example, because it can fail at random place, here - then publishing the artifacts):
...
[14:21:23]Publishing artifacts (running for 5m:15s)
[14:21:23][Publishing artifacts] Collecting files to publish: [Output/Blendr/iphoneos/Artifacts/*, Output/Blendr/iphoneos/Artifacts/Distribution/*]
[14:21:24][Publishing artifacts] Sending Output/Blendr/iphoneos/Artifacts/*
Please help us!
And many thanks for your great product anyway!
Please sign in to leave a comment.
Hi Anton,
Please attach few agent thread dump while the issue is reproduced.
Thanks, Alina.
Will dump, then reproduced next time.
Here is the dump:
https://gist.github.com/yakimant/de674d3b72d5fc895d5a
Hi,
Thanks for log. Unfortunately it is not possible to found out the root of the problem from one log, we still need few (for example 5-10) agent thread dumps with 5-10 seconds interval. Also please add “-l” option to jstack command to include additional information about locks. It would be great if you have a chance to take such sets of thread dumps when build hangs in different phases.