Mac OS X agents: hang up

Hello Jetbrains and TC community!

Our company has issues for Mac OS X. They are hanging up periodicaly in different phase of build proccess.
If I press "Stop" for the hanging build - nothing happens.
If I make a limitation for successful build - it will not be failed after time limit.
We don't know, what to do and how to fix or debug it.
If we don't watch the agents page on TC server - we have builds hanging up for 30 hours and more.

Here is some (poor) information:
1) "View thread dump" on TC server:
TeamCity was unable to locate any processes of this build.

2) TC agent logs:
Paulie-Pennino:logs qa$ ls -asl | grep "2 Jul 15:"
    0 drwxr-xr-x  18 qa  staff       612  2 Jul 15:18 .
10112 -rw-r--r--   1 qa  staff   5176851  2 Jul 15:21 output.log
5848 -rw-r--r--   1 qa  staff   2990096  2 Jul 15:25 teamcity-agent.log
16688 -rw-r--r--   1 qa  staff   8543728  2 Jul 15:21 teamcity-build.log
20488 -rw-r--r--   1 qa  staff  10485828  2 Jul 15:18 teamcity-build.log.1
2880 -rw-r--r--   1 qa  staff   1474473  2 Jul 15:05 teamcity-vcs.log
Paulie-Pennino:logs qa$ tail -10 teamcity-agent.log
[2014-07-02 15:21:28,407]   INFO -    jetbrains.buildServer.AGENT - Updating agent parameters on the server: AgentDetails{Name='Paulie Pennino', AgentId=42, BuildId=190692, AgentOwnAddress='null', AlternativeAddresses=[192.168.21.94], Port=9090, Version='27767', PluginsVersion='27767-md5-57a86f4200ced1f3112dc533439269b7', AvailableRunners=[Ant, Duplicator, gradle-runner, Inspection, Ipr, JPS, Maven2, rake-runner, simpleRunner, Xcode], AvailableVcs=[cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='c89b1691a521b700cecac0a356e29100'}
[2014-07-02 15:25:41,179]   INFO - s.agent.StacktracesAgentPlugin - Start getting thread dumps
[2014-07-02 15:25:41,179]   INFO - rocesses.ProcessTreeTerminator - Using jetbrains.buildServer.processes.ProcessTreeTerminatorMac
[2014-07-02 15:25:41,227]   INFO - stacktraces.agent.DoThreadDump - Running processes:
[2014-07-02 15:25:41,227]   INFO - acktraces.agent.nix.JdkLocator - Looking for JPS in agent-running jvm...
[2014-07-02 15:25:41,232]   INFO - acktraces.agent.nix.JdkLocator - Lookup for JPS. Found: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/jps
[2014-07-02 15:25:41,232]   INFO - r.stacktraces.agent.nix.JpsRun - Fetcing list of running java processes
[2014-07-02 15:25:41,232]   INFO - r.stacktraces.agent.nix.JpsRun - Starting jps: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/jps -q
[2014-07-02 15:25:42,016]   INFO - r.stacktraces.agent.nix.JpsRun - Found java processes: [66673]
[2014-07-02 15:25:42,016]   INFO - aces.agent.nix.NixDoThreadDump - detected java processes: [66673]

3) TC build log (as example, because it can fail at random place, here - then publishing the artifacts):
...
[14:21:23]Publishing artifacts (running for 5m:15s)
[14:21:23][Publishing artifacts] Collecting files to publish: [Output/Blendr/iphoneos/Artifacts/*, Output/Blendr/iphoneos/Artifacts/Distribution/*]
[14:21:24][Publishing artifacts] Sending Output/Blendr/iphoneos/Artifacts/*


Please help us!
And many thanks for your great product anyway!

4 comments
Comment actions Permalink

Hi Anton,

Please attach few agent thread dump while the issue is reproduced.

0
Comment actions Permalink

Thanks, Alina.
Will dump, then reproduced next time.

0
Comment actions Permalink

Hi,


Thanks for log. Unfortunately it is not possible to found out the root of the problem from one log, we still need few (for example 5-10) agent thread dumps with 5-10 seconds interval. Also please add “-l” option to jstack command to include additional information about locks. It would be great if you have a chance to take such sets of thread dumps when build hangs in different phases.

0

Please sign in to leave a comment.