Server vs agent builds

Our configuration:
All machines running on 64-bit CentOS release 5.5 (Final) VMs
5 active build agents - one agent running on the server
All machines running 64-bit JDK 1.6.0.19
Builds are either maven or shell script-based.

Problem: There is a huge difference in build time between running a build on the agent that runs on server vs the one that runs on separate VMs. Builds that run on an external agent take at least twice as long and in some cases 3-4 times as long to run builds that run on the agent that is on the server.

On the server:
TEAMCITY_SERVER_MEM_OPTS="-Xms1536m -Xmx1536m -XX:MaxPermSize=512m"
TEAMCITY_AGENT_MEM_OPTS="-Xmx384m"

On each of the external agents:
TEAMCITY_AGENT_MEM_OPTS="-Xmx1024m"

We recently added 2 vCPUs to each build machine in the hope that this will shorten build times but it only shortened the build time of the builds running on the server's agent.

Do you have any suggestions of how to reduce build time on the external build agents?

Thanks,
-Dave

11 comments

Hello Dave,

Could you please be more specific and describe what do you meant by "in some cases 3-4 times as long to run builds"? How long does it takes in both cases?
You can take a look into build log and find information about state in which the build arrives longer comparing to the build running on another agent. Please provide these build logs.

Kind regards,
Marina

0

Marina,

I thought I had attached this image to illustrate the problem:

Screen shot 2010-10-05 at 2.54.37 PM.png

This is a regular remote run build that developers tend to run through IntelliJ. Note the build duration differences between the tcagents and the tcserver.

-Dave

0

Dave,

We still need these build logs. Also can you please tell why do you need to increase memory for external agents? Was there any problems before you did that?

Regards,
Marina

0

We were having memory errors at times - perhaps because we are using both a 64-bit OS and a 64-bit JVM. Memory problems tended to happen when remote deployment builds were running groovy-based tests.

Do you want the build agent logs? If so, do you want the logs from all of the build agents or one set from one TeamCity agent and one set from the Teamcity server's agent?

0

I meant "build log": available at build result page. The build log as result of  running build on tcagent agent and the log after running on the tcserver agent.

0

Sorry for not getting back to you sooner.

I have attached a zip file containing 2 log files. Each build used the same build config and the build config is used as a remote run. The one called tcserver.log was a build that took 6:33. The one called tcagent03 took 18:29.

-Dave



Attachment(s):
build_logs.zip
0

Dave,

The timings in the log seem to be distributed evenly through the build with clear indication of disk operations on tcagent03 being about twice longer then on tcserver agent. So far no reasons to blame TeamCity.

I'd appreciate if you can run the same build on both agents without TeamCity to make it clear weather it is TeamCity who introduces the increase in time or just different agent "hardware" resources.

0

I started the same process on the tcserver & tcagent03: 'mvn -q clean install' from a command line.

The build time results are as follows:

tcserver                     tcagent03
6 min 40 seconds     10 min 23 seconds

So, there is a difference. Not as much as running the build through TeamCity but I have a theory about that.
This may have to do with a configuration difference that causes checkouts to take longer.

The tcserver & the subversion machine are set to UTC.
The external build agents are set to EDT.

The obvious "hardware" differences between the machines is that the external agents have 2GB of memory and the tcserver has 4GB.

I will see about making all the machines more uniform.

Thanks for your time.

-Dave

0

Dave,

This may have to do with a configuration difference that causes checkouts to take longer.

The tcserver & the subversion machine are set to UTC.
The external build agents are set to EDT.


So far I am not sure timezone differences can influence performance. What can is bandwidth between the machines and given that they are located in different timezones (so probably many kilometers in between) this can be an issue.

0

In this case, the machines are actually located in the same building. They were just had their time zones configured differently.

0

Dave,

So far I cannot imagine how timezone difference can affect actual time spent by the build checking out sources or building (other then affect reported times).

If you figure out what the real cause it or whether there is something to fix in TeamCity - please let us know!

0

Please sign in to leave a comment.