Monstrously slow builds

I'm attempting to migrate from CruiseControl.NET to Team City. Perforce check-out takes over 40 minutes. Doing it manually using P4Win takes 5. (Doing it on my PC, not in a VM takes 1). Once the build has actually started, if I'm lucky it takes 20 minutes. Logging on to the build agent, I see java.exe taking as much as 99% of the time, with msbuild and csc only ever hitting 30% at most.

I took a guess that its just spamming java with output so I tried switching to MSBuild (from Sln2005) and doing /verbosity:quiet, but this doesn't affect the output at all.

My setup:

I have two windows 2003 server VMs sharing a Quad-Core 2008 x64 Hyper-V host with 8Gb ram. One server is the web server, the other is a build agent. They communicate using a private VM network - the build agent has "ownaddress" set as well as "ownport". The web server has 768Mb RAM, the build machine 2048Mb. Our CC.NET build agents also have 2048Mb ram. The Virtual Hard disks of the two VMs are on separate physical disks.

I'm not sure that there is a solution. The perforce check-out time is something of a showstopper, but I hope the build agent speed could be improved.

Jamie

6 comments

For reference, I just installed CC.NET on BuildAgent VM machine, and it (CC.NET) took under 5 minutes to do a clean get from perforce, and 7 minutes to do a full build. Additionally, the CC.NET perforce checkout included 1.92Gb of binary data files that I have excluded from Team City's client spec for speed reasons. :(

Jamie

Edited by: James Briant on Mar 19, 2008 11:12 PM

0

I am having the same problem with TeamCity/CVS, with TC running on a Solaris box, virtual server (everything else on it runs just fine.) Doing Maven builds from the command line takes no more than a minute, while TC takes up to 20 minutes. So far, it has been absolutely unusable, and we are putting it on the shelf, unfortunately. Can't figure out what the problem is. Anyone from JetBrains have any idea what may be causing such poor performance? P-lease???

0

Thanks for the reply. Here is the CVS configuration detail:

VCS root name: My project
Type of VCS: CVS
Module name: MyProjectModuleName (as in CVS)
CVS root: :ssh:myname@ipAddress:2401/home/cvs (where ipaddress is: xxx.xxx.xxx.xxx)
x Checkout HEAD revision
Quiet period: 3
SSH port : 22
SSH password: .....mypassword to cvs
checking interval: 30


Actually the behavior of "Checking for changes" changes. Sometime it takes around 10 minutes before maven build; other times it takes forever.

If you give me your email address I can attach a screenshot to you. Feel free if you need more information for the investigation such as the agent log and config files, etc.

Susan

0

Hello,

First of all, please try to install build agent on different machine than TeamCity. TeamCity definetely has overhead when build is running to provide ongoing UI feedback, reports etc. This includes sending build messages from the build to server, writing logs, updating database.

Second, please check how much output your build generates. As long as this output is logged by TeamCity, it results in additional overhead.

Third, please check build agent logs at buildAgent/log directory (teamcity-agent.log). It has timings and you may get a clue which steps of your build take most of the time.

And, we would really appreciate if you share your findings with this problem.

Kind regards,
KIR

0

James,

Sorry for the delay in replying.

Unless your build produces enormous output, problem with too long builds is quite strange and requires careful investigation.

For a start, could you please share your build logs?

You can create an issue in our tracker: http://jetbrains.net/jira/browse/TW and attach it there, or (for a large archive or to keep it private) you can upload your build log to ftp://ftp.intellij.net/.uploads and let us know the exact file name.

If your CC build log has timestamps in it, could you also provide it?

We would appreciate if you can help us investigate and pinpoint the issue.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

If you're using CVS, it could be the first step "checking for changes" is taking the majority of the build. What does it look like it happening for most of those 20min?

In my case, the suggestion provided to me here: http://www.intellij.net/forums/thread.jspa?threadID=274187&tstart=30 helped.
The gist of it is to set up an artifact dependency on another configuration that has already checked out locations that do not change.

Edited by: aebaugh on Apr 3, 2008 5:11 AM

Edited by: aebaugh on Apr 3, 2008 5:13 AM

0

Please sign in to leave a comment.