Best least-impact solution to binary references in VCS
Hello all,
We are using TeamCity 2017.1 and have been using it for years with great joy. A long time ago, someone decided that all third-party binaires should be put into Subversion (our VCS of choice).
This has worked fine, but over time this repos has grown quite large, and combined with our being better and better at using TeamCity, we now have dozens of build configurations which all uses third-party binaries.
Our third-party folder is called Department and is around 2.6 GB in size. As such this is not so bad, but remember that this folder is used by pretty much every single project on the build server!
Now, I will agree with everyone that says that we should use Nugets, network shares etc., and that would work great with new projects. However, we have a lot of history and we cannot begin to change every single solution and branch.
A co-worker came up with the idea, that IF we made a single build project that in reality did nothing but keep a single folder updated with our Department stuff. Then we just need to find a way to reference this, without have to change all our projects and solutions.
My initial though is using Snapshot dependencies and then create a symbolic link as the first build step and remove it as the last, in order to achieve the same relative levels.
But is there a better way? What do other people do?
And keep in mind, that replacing with nugets or something else is not an option.
Please sign in to leave a comment.
I posted this here as well: https://stackoverflow.com/questions/46175651/least-impact-solution-to-binary-references-in-vcs
Hi Thomas,
this would definitely be challenging, and the link idea is probably the only short-term solution without modifying your pre-existing builds. Obviously, I'd suggest first to reevaluate your approach, and once you decide, I would definitely start migrating all your projects (also existing) to the new format. It might be easier to do if you activate versioned settings, as you would be able to perform several of the modifications through a script.
Versioned settings, particularly in Kotlin, would help immensely by making it very, very easy to wrap any build configuration with the two build steps you want to add (adding the link and removing it), but this would still mean modifying the current configurations.
2.6 GB doesn't sound like too much to store on an agent permanently, but does sound like something that would slow down builds a lot with having to transfer them through the network all the time.
Without modifying the actual configurations, the only other option I can think off would be to create, in every agent, a copy of the folder in the place where your application would expect to find it (or elsewhere and link it there). Then you can create a build configuration that, when the content of the folder is updated, updates it in every build agent (probably via a script).
I would argue though that you should move away from your current status as soon as possible, maybe duplicating your build configurations (extract them as a template, then it'll be very easy to duplicate them, simply remove the VCS root from the second), adding some sort of "legacy" tag to the previous model and starting ASAP with a new, scalable model, be it nuget, network shares the artifact dependency option we provide in teamcity (you could have for example a build configuration that publishes all binaries as artifacts, and you could pick them up from any build configuration by referencing them there), or any other option you consider appropriate.
Hope this helps.