Very slow gradle build created from a template

Hi,

I am attempting to use Teamcity templates so that users in my organization can easily create new builds that share most processes, based off a template.

For some reason, builds generated using the template run *extremely* slowly. I've looked at the server and agent logs, and they appear to indicate that every once in awhile, after a BlockClosed statement is issued to close a block in the log, nothing happens for a few minutes. For example:

[2013-12-10 13:13:19,510] out - ##teamcity[blockOpened name=':work-i18n-impl:jar']

[2013-12-10 13:13:19,576] out - ##teamcity[blockClosed name=':work-i18n-impl:jar']

[2013-12-10 13:14:08,380] out - ##teamcity[compilationStarted compiler=':work-motivate:compileJava']

[2013-12-10 13:14:08,492] out - ##teamcity[compilationFinished compiler=':work-motivate:compileJava']

This is also apparent in the build output:

[13:13:19][Step 1/6] :work-i18n-impl:jar

[13:14:08][Step 1/6] :work-motivate:compileJava


It appears that for some reason, nothing happens for nearly a minute in between closing a block and opening the next one. The end result is that this build, which other than being created from a template is identical to our other builds, takes hours to run instead of minutes.

The build is running gradle. I have attached two thread dumps from the agent. We are running Teamcity 8.0.3. Any help would be appreciated.

Thanks!


Mike Shea.


Attachment(s):
agentDetails (1).html.zip
agentDetails.html.zip
7 comments
Comment actions Permalink

Is it possible that JVM running this build script does not have enough memory? I'd also try to take thread dump (see link "View thread dump" on build results page) while build is in hanging state.

0
Comment actions Permalink

Hi Pavel -

Thanks for your reply. I have attached two thread dumps, taken ~40 seconds apart, both during the same "blocked" period of the build. Please let me know what you think.



Attachment(s):
thead_dump_2.txt.zip
thread_dump_1.txt.zip
0
Comment actions Permalink

Have you tried to run this build script manually from command line on the same agent?

0
Comment actions Permalink

Hi Pavel -

Yes, I have tried doing that. It runs at normal speed when I run it from the commandline, and completes in a few minutes rather than in an hour and a half.

0
Comment actions Permalink

Did you run it under the same user and on the same machine where agent is installed? I'd also try to set the same environment variables that are set for the build script.
When agent starts a build there is no any magic, it's just start a command line and that's it. So most likely the difference is in environment.

0
Comment actions Permalink

Hi Pavel -

Yes, I ran it on the same agent and also yes, using the same user. I ran with the same commandline parameters, but it is possible that some environment variables are having an impact.

My initial hypothesis is that something related to the logging is causing logging output to block during the hanging periods. I do not know how the logging actually occurs in the context of Teamcity (whereas normally when I run a build, stdout is just my terminal or a file that I pipe it to).

I will work on getting the environment as close as possible to what Teamcity is doing and report back.

0
Comment actions Permalink

TeamCity gradle listener outputs service messages to stdout. AFAIK, there are no network operations involved, so it should not affect build anyhow.

0

Please sign in to leave a comment.