Mac OS X agents: hang up

Hello Jetbrains and TC community!

Our company has issues for Mac OS X. They are hanging up periodicaly in different phase of build proccess.
If I press "Stop" for the hanging build - nothing happens.
If I make a limitation for successful build - it will not be failed after time limit.
We don't know, what to do and how to fix or debug it.
If we don't watch the agents page on TC server - we have builds hanging up for 30 hours and more.

Here is some (poor) information:
1) "View thread dump" on TC server:
TeamCity was unable to locate any processes of this build.

2) TC agent logs:
Paulie-Pennino:logs qa$ ls -asl | grep "2 Jul 15:"
    0 drwxr-xr-x  18 qa  staff       612  2 Jul 15:18 .
10112 -rw-r--r--   1 qa  staff   5176851  2 Jul 15:21 output.log
5848 -rw-r--r--   1 qa  staff   2990096  2 Jul 15:25 teamcity-agent.log
16688 -rw-r--r--   1 qa  staff   8543728  2 Jul 15:21 teamcity-build.log
20488 -rw-r--r--   1 qa  staff  10485828  2 Jul 15:18 teamcity-build.log.1
2880 -rw-r--r--   1 qa  staff   1474473  2 Jul 15:05 teamcity-vcs.log
Paulie-Pennino:logs qa$ tail -10 teamcity-agent.log
[2014-07-02 15:21:28,407]   INFO -    jetbrains.buildServer.AGENT - Updating agent parameters on the server: AgentDetails{Name='Paulie Pennino', AgentId=42, BuildId=190692, AgentOwnAddress='null', AlternativeAddresses=[], Port=9090, Version='27767', PluginsVersion='27767-md5-57a86f4200ced1f3112dc533439269b7', AvailableRunners=[Ant, Duplicator, gradle-runner, Inspection, Ipr, JPS, Maven2, rake-runner, simpleRunner, Xcode], AvailableVcs=[cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='c89b1691a521b700cecac0a356e29100'}
[2014-07-02 15:25:41,179]   INFO - s.agent.StacktracesAgentPlugin - Start getting thread dumps
[2014-07-02 15:25:41,179]   INFO - rocesses.ProcessTreeTerminator - Using jetbrains.buildServer.processes.ProcessTreeTerminatorMac
[2014-07-02 15:25:41,227]   INFO - stacktraces.agent.DoThreadDump - Running processes:
[2014-07-02 15:25:41,227]   INFO - acktraces.agent.nix.JdkLocator - Looking for JPS in agent-running jvm...
[2014-07-02 15:25:41,232]   INFO - acktraces.agent.nix.JdkLocator - Lookup for JPS. Found: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/jps
[2014-07-02 15:25:41,232]   INFO - r.stacktraces.agent.nix.JpsRun - Fetcing list of running java processes
[2014-07-02 15:25:41,232]   INFO - r.stacktraces.agent.nix.JpsRun - Starting jps: /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/jps -q
[2014-07-02 15:25:42,016]   INFO - r.stacktraces.agent.nix.JpsRun - Found java processes: [66673]
[2014-07-02 15:25:42,016]   INFO - aces.agent.nix.NixDoThreadDump - detected java processes: [66673]

3) TC build log (as example, because it can fail at random place, here - then publishing the artifacts):
[14:21:23]Publishing artifacts (running for 5m:15s)
[14:21:23][Publishing artifacts] Collecting files to publish: [Output/Blendr/iphoneos/Artifacts/*, Output/Blendr/iphoneos/Artifacts/Distribution/*]
[14:21:24][Publishing artifacts] Sending Output/Blendr/iphoneos/Artifacts/*

Please help us!
And many thanks for your great product anyway!

Comment actions Permalink

Hi Anton,

Please attach few agent thread dump while the issue is reproduced.

Comment actions Permalink

Thanks, Alina.
Will dump, then reproduced next time.

Comment actions Permalink


Thanks for log. Unfortunately it is not possible to found out the root of the problem from one log, we still need few (for example 5-10) agent thread dumps with 5-10 seconds interval. Also please add “-l” option to jstack command to include additional information about locks. It would be great if you have a chance to take such sets of thread dumps when build hangs in different phases.


Please sign in to leave a comment.