concurrent msbuild builds
Suppose i have 8 c# solutions with 3 layers of dependencies in between.
I need to start compiling 3 solutions in parallel.
Next 2 will depend on one of the pre-compiled 3 and will compile in parallel.
Next 3 will depend on one of the pre-compiled 2 and will compile in parallel.
All checked-out (got) sources are shared (come from single tfs branch/folder).
What is the exact tc configuration I should follow i.e.
I understand i should create multiple agents.
Do i need a composite build configuration to check-out the sources only once?
How do i manage build and snapshot dependencies to reuse the sources?
Please sign in to leave a comment.
Hi Alexander,
in your situation, the set up would be the following:
-Have 3 agents, as 3 is the maximum amount of tasks you need to run in parallel. Extra agents would allow you to run even more, but for only this project wouldn't be needed. You can install them in separate machines or in the same one.
-Have 8 build configurations, each with a C# solution. If configuration is the same or similar, you can use a template to reduce copying configs from one to other.
-Have the last 3 depend on the intermediate 2, and those 2 depend on the first 3. That way, triggering the final 3 builds will push the rest to the queue, and the intermediate two will wait to the initial ones, then the final ones will wait to the intermediate.
With this setup, if you want to use the same folder, you can use the "checkout directory" parameter of the version control settings to fix a folder for all builds, but this might lead to issues, as three different processes would try to get access to the same files. Further, some builds might decide that the folder is not clean and thus simply clean it from all artifacts and perform a clean checkout. It would also lead to issues in case any artifact or old file remains and it would lead to the failing of a build, it will likely fail the three builds making it much harder to trace.
You say that all 8 solutions are in the same branch, but can't they be in separate folders? If they were in separate folders, then you could easily use checkout rules so each build configuration only checks out the part that it needs, and nothing else. Simply leave the default folder for work, use checkout rules so only the required part is checked, and use artifacts and artifact dependencies if you need some files or packages from one build to the next one. This is the recommended approach. While you would need a bit of extra time as each checkout would be done separately, it would allow for much greater flexibility.
Another possibility would be to remove the VCS Root from all builds, create an initial empty build that only performs the checkout, then have your first 3 solutions depend on that one and use it as "checkout directory". As no VCS is on them, the checkout directory will be used as a work directory, and the start of the build will not try to perform a checkout, thus shouldn't remove the contents, but with 3 solutions trying to access the same files, this might lead again to issues with concurrent access to files, so it's not recommended.