How to make snapshot dependency run in parallel for faster builds
Our project is compiled for multiple platforms. During the build the data needs to be processed and the source code needs to be compiled. This takes approximately 7 hours per platform.
Currently we sync from perforce in a first build configuration (checkout to custom folder) and we execute processing to generate project solution etc. The processing takes approximately 5 mins on average but a clean perforce checkout can take 4-5 hours due to its size.
During the weekend we clean the perforce folder, but on working days perforce is just syncing incrementally. So the first build configuration takes between 5-10 mins on average.
Due to the long build times we want to add additional build agents. Although the system above has worked well in the past on a single agent, it doesn't allow for parallel execution over multiple build agents. We could dedicate a single build agent per platform but this feels like a waste of resources.
If the first build configuration is A, we have configurations B, C, D and E which all depend on A. It is possible to launch builds manually on the other agents, but the automatic triggering will result in all the builds being queued up on the first available agent (the one that has been assigned A). I've tried to change settings but nothing seems to be doing what we want. We would basically want to tell A-B, A-C, A-D and A-E are build sets and overall it is a lot faster to assign A to multiple build agents.
Basically what is 'wrong' in our setup is that the perforce checkout A should be part of B, C, D and E but that would mean that each build agent wants to create 4 perforce workspaces (one for each platform) per agent. If we want to clean all that would take 4x 4-5 hours per clean perforce checkout.
Any suggestions?
Thanks
Please sign in to leave a comment.
If I'm understanding you correctly, you have four builds (B, C, D and E) which are dependent on A. You wish to have B, C, D and E all dependent on A. With a single build agent, the builds are queued up and eventually finish but it takes a long time. By adding additional agents, you could have B, C, D and E running in parallel to improve the total build time, which is true. But I'm not understanding how this would be wasteful with resources, can you elaborate on that?
Build A would run a single time and B, C, D and E would use that snapshot dependency to run after A completes. If you had four build agents, all four builds could run at the same time with the same snapshot of A. Please see our documentation on the concept of Build Chains to see if this would fit your needs. You might also find the documentation on Build Dependencies helpful, there is a lot of detail on the different use cases in there.
If you always want to have B, C, D and E run every time build A completes, you could kick off the whole build chain with a single Composite Build that is dependent on the four builds. This would ensure everything is queued up in the right order. If you're interested in learning more about composite builds, please review our documentation available here.
I probably didnt explain it well :(
A is checking out to a (fixed) custom folder. B, C, D and E rely on the folder being there and being preprocessed by A. This implies that B, C, D and E should run on the same agent as A. It would be faster if A is being 'build' multiple times on the build agents so B, C, D and E can run in parallel. Hope that is more clear?
If B, C, D and E would check out their own copy of the same project it would take very long for a clean perforce checkout. Unless configuring B, C, D and E to check out to the same folder force them to use the same perforce workspace?
I think I'm following and this could be accomplished in a few ways, depending on what builds B, C, D and E are doing. Probably the first thing you could do would be to checkout build A to a commonly available location. This could be either a network location that all four agents can access or maybe you have all four agents running on the same machine. Regardless, it would only need to be checked out a single time to a single location. Then it is just a matter of what all aspects are required from A; is it just the artifacts they are after or is it dependent on the build as a whole? You can use a combination of artifact dependencies and snapshot dependencies to set up your build chain.
A is basically a check out to a custom folder (on the build agent) and data processing of multiple Gbs of data. B, C, D and E rely on ALL of the contents of that folder to be present. B, C, D and E need A to have happened (so all the contents are ready for building). A must be local because on a network share it will slow down B, C, D and E too much.
In principle A is part of B, C, D and E. Since it is pretty big A was separated. However in case of multiple build agents it would be faster for each build agent to have a copy of A. If B, C would happen to run on the same build agent they should share A.
Does that make more sense?
If using a shared drive for build A checkout isn't an option and you don't want to transfer the build A contents from one machine to another after build A is completed, I think you're probably limited to two options.
Hello Eric,
How do I use build A as a template for A1 A2 A3 and A4? Does this imply A1 and A2 will checkout A twice if they happen to run on the same agent?
Thanks
Instructions for extracting a build template from an existing build configuration can be found in our documentation here: https://www.jetbrains.com/help/teamcity/build-configuration-template.html#BuildConfigurationTemplate-Creatingbuildconfigurationtemplate
As I mentioned above, you would be able to have each of BCDE run independently of one another, but each will checkout build A. A1, A2, A3 and A4 would all technically be separate build configs and would each need to be checked out. However, making changes to all four would be accomplished by editing the single template.