Why is it taking a long time to resolve my artifact dependencies?

In my project I have one configuration that builds a web page and published the artifacts.  about 120 MB in less than 6000 files.  I manually copy these archives from the main machine to the agent in less than a minute, but during build of my installer configuration which pulls all of the artifacts via a "**" wildcard it takes more than 30 minutes.

Anyone have an idea what's going on here?

7 comments
Comment actions Permalink

Operation can take much time if number of downloaded artifacts is huge. Probably it will be faster to pack files in a zip and then use this zip file in artifact dependency.

0
Comment actions Permalink

What?  Is it really supposed to take 30 times longer to move the artifacts from a dependent configuration to the one building than it does when I copy the files via normal file system?

I know there is a API of sorts that we can use to do custom code, to use the file system to grab the artifacts, but surely the 'out of the box' product should act faster than this, right?

If the answer is that 'out of the box' it is designed to be this slow, and the thing that will make it faster is to do this zip proceedure, how exactly does that work?  Is there a setting somewhere that will zip all the files as the configuration is built?  How would I then say I don't really want the zip file in my configuration as I grab the zip artifact - I actually want it decompressed?

Thanks

0
Comment actions Permalink

It would be much more productive to provide some details on your setup. How your artifact dependencies are configured, can you provide a screenshot? Can you provide a build log?

Comparing to file system is not fair at all just because TeamCity delivers artifacts by HTTP protocol, and if you have more than a thousand of files to download then it means that more than a thousand of HTTP requests will be sent to the server. That is why packing the files to a zip archive will work faster.

0
Comment actions Permalink

Here's a snipit from a successful build log - notice the 30 minute gap.

[10:02:51]: [Resolving artifact dependencies] Artifact Distrib/UltimateSoftware.Services.dll downloaded from <R2 - UltiPro.NET Core :: 1 - Build Ultipro.Net Core (build #892)> saved to: C:\BuildAgent\work\b0f9544cc6925f2d\UltiPro.NET\Distrib\UltimateSoftware.Services.dll
[10:02:51]: [Resolving artifact dependencies] Artifact Distrib/UltimateSoftware.SiteInfo.dll downloaded from <R2 - UltiPro.NET Core :: 1 - Build Ultipro.Net Core (build #892)> saved to: C:\BuildAgent\work\b0f9544cc6925f2d\UltiPro.NET\Distrib\UltimateSoftware.SiteInfo.dll
[10:02:51]: [Resolving artifact dependencies] Artifact Distrib/UltimateSoftware.Security.Base.dll downloaded from <R2 - UltiPro.NET Core :: 1 - Build Ultipro.Net Core (build #892)> saved to: C:\BuildAgent\work\b0f9544cc6925f2d\UltiPro.NET\Distrib\UltimateSoftware.Security.Base.dll
[10:32:59]: [Resolving artifact dependencies] Artifact UltiPro.Net/WebSites/UltiProNet/Bin/mappings/TaxRate.xml downloaded from <R2 - UltiPro.Net Website :: Build Ultipro.Net Website (build #872)> saved to: C:\BuildAgent\work\b0f9544cc6925f2d\UltiPro.Net\WebSites\UltiProNet\Bin\mappings\TaxRate.xml
[10:32:59]: [Resolving artifact dependencies] Artifact UltiPro.Net/WebSites/UltiProNet/pages/edit/PMPerformanceReview.aspx downloaded from <R2 - UltiPro.Net Website :: Build Ultipro.Net Website (build #872)> saved to: C:\BuildAgent\work\b0f9544cc6925f2d\UltiPro.Net\WebSites\UltiProNet\pages\edit\PMPerformanceReview.aspx
[10:32:59]: [Resolving artifact dependencies] Artifact UltiPro.Net/WebSites/UltiProNet/stylesheets/jqGrid/coffee/grid.css downloaded from <R2 - UltiPro.Net Website :: Build Ultipro.Net Website (build #872)> saved to: C:\BuildAgent\work\b0f9544cc6925f2d\UltiPro.Net\WebSites\UltiProNet\stylesheets\jqGrid\coffee\grid.css


I have also included a screen snipit of the dependency that pulls in these files.

OK - so we're locked into HTTP protocol - how exactly can we get the files to automatically be zipped and unzipped?  Is there a setting that I am missing within the UI?


Attachment(s):
snipit.JPG
0
Comment actions Permalink

I see the gap, and if there were no other lines it looks strange. What is the size of UltiPro.Net/WebSites/UltiProNet/Bin/mappings/TaxRate.xml ?

Automated zip file creation will be available in TeamCity 5.1, right now the only way is to add some code to your build script to zip artifacts before build finishes. But lets find a real cause of this problem before you will go this way.

0
Comment actions Permalink

I should have included a larger snippit of the dependencies.  (which is now attached)  The .dll is coming from a different dependency.  The .xml file is just the first (very small) file in a list of about 2000 files.  It seems that there is processing done to gather all of the file data - then pushed into the log file.  - (wheter the files are copying before the log shows it I cannot tell)
We can probably live with the time gap until 5.1 as long as it's not too far out.  Do you have an estimate of the release time frame for that?

Thanks!



Attachment(s):
snipit.JPG
0
Comment actions Permalink

We plan to publish the first EAP of 5.1 during this week. This EAP will have automatic artifacts archiving feature. 5.1 was planned to be released in March, but it is more likely to happen in April.

0

Please sign in to leave a comment.