Sharing "large" dependencies/NuGet packages between builds
I tried many different solutions and also browsed various forums, but still struggling to find any solution which would fully satisfy our needs - there are always some gotchas involved. I would appreciate if anyone could share their experience or best-practice how to achieve good results.
We have quite big build in TeamCity, which we distributed into separate smaller builds - to be able to parallelize some long running operations, and to follow "build once, deploy anywhere" rule. Our project was at first .NET Framework 4.6+, but now we migrated to .NET Core 2.0, and build became even more slow (because of "distributed" .NET Core nature).
Main problem is related to NuGet packages, and it comes from the fact, that in our environment we have around 20-30 agents (mainly EC2 virtual agents spawned automatically on demand). It means that we cannot rely on previously restored NuGet packages - there just can be no packages on newly spawned agent.
It wouldn't be a problem if we would use one big build, executed on same agent, with no parallelization - we would need to restore NuGet only once. However now we need either to share NuGets as dependency artifacts or just to restore all NuGets on all related builds. Neither way seems optimal. Producing/publishing artifacts for all NuGet packages takes almost same time as restoring them, while restoring just takes 3-5 minutes. Ideally I would like to restore just once per solution, but really not sure how to achieve it..
I isolated and checked only needed NuGet packages for our .NET Core 2.0 project - and it's 800-900MB in size. Would it be a good idea to create such big artifact? I see it's really not perfect solution in terms of time needed, but maybe it's common practice? Maybe there are some other top-secret solutions, like restoring shared NuGet packages to some sort shared network location, or smth?
Actually it's not a problem solely related to NuGet/.NET Core, because I already experienced something similar with NPM packages as well.
Please sign in to leave a comment.
Hi Aleksandras,
This usually depends on your infrastructure and the best suggestion is that you run several tests on your own environment so that you can figure out what works best for you.
The solutions are mostly what you described:
-Use NuGet feeds (TeamCity can act as one) and restore whenever needed
-Use artifact dependencies
-Use a network share that is mounted as the first step of the build
A usual scenario here is caching artifacts in the agent, so they might take a while on the first run but are much faster on subsequent ones. This obviously won't work for EC2 images, so the three points mentioned above are the only options I can currently recommend.
The network share option is often used for static, large resources. Otherwise, you need to ensure that code/packages are up to date.
Obviously, depending on your project structure, there might be ways to reorganize it in such a way that transfer times and sizes are reduced, trying to get builds to only use a subgroup of the dependencies, but this is 100% dependent on your project, so no possible suggestions from our side there.