Re-using same build chain for multiple target platforms alternatively
I have a project containing approx. 20 different sub-projects. Each sub-project is represented as a separate build configuration in that project and can be build separately, producing each its own artifacts (a feature which is used quite frequently). Several times daily the complete project needs to be build. For this I have created another build configuration in the same project called "build_all_windows". By defining snapshot dependencies with all other sub-projects it triggers re-builds of the sub-projects (only) if needed. It collects all results, plus there are some additional build steps required for deployment purposes. I use the same VCS for all projects with the option "run build on the same agent", because some projects later in the build chain require not only the same files but some scripting results of prior steps. Overall, I think this works quite well. The dependencies between the sub-projects are all defined as snapshot dependencies.
Now to my problem: I can't make this setup work when I add the requirement to compile this whole project optionally for a different target platform alternatively. My first solution was to have 2 distinct agent pools ("windows", "linux") and create another build configuration "build_all_linux". The new build configuration has the "agent equals linux" requirement, the old one having likewise the requirement for "windows" agents. All sub-projects can still be build with all agents, whether it be linux or windows. The individual build steps of the sub-projects know what the target platform is (%system.agent.work.name% etc.) and make any platform differentiation there (if needed at all).
My problem is, even if adding artefact dependencies to the existing snapshot dependencies, TC seems not to be able to apply the agent requirement of the last step in the build chain (my "build_all_xx"-build configuration) to other build configurations in the build chain. So if my last build was a windows build and all sub-projects are compiled for the windows platform, the "build_all_linux" won't work now because it thinks all dependencies are already build. But they are for the wrong platform only. This leads either to a "no agent"-entry idling in the build queue if no agent was specified for the "build_all"-BC or an error message in the build steps because the work on the correct platform was not done.
I only see two solutions:
- Either to duplicate all 20+ sub-projects for each target platform, resulting in much more maintenance work (each build step change or version jump will need to be done twice), even though all (complex) build steps remain totally the same. Then I have a separate sub-project for each platform and my "build_all"-BC depends on the sub-projects with the correct target platform only.
- Or I could require that for every build everything must be done from scratch again (resulting in much more compilation time, most of the time without need, since the platform may remain the same between two runs). This way each build would cost hours.
For me both options seem to be quite sub-optimal. I am sure there must be a better way to do this. How can I swap between multiple platforms without any time-consuming "rebuild all" or a senseless duplication / maintenance hell of having many build configurations?
Thanks
Ray
Please sign in to leave a comment.
You can override parameters from dependencies: https://confluence.jetbrains.com/display/TCD18/Predefined+Build+Parameters#PredefinedBuildParameters-OverridingDependenciesProperties
This would allow you to force the parameter into the dependencies, but it would rebuild them as they wouldn't consider their last runs suitable, because you are forcing a parameter change.
With this in mind, the best option would be to duplicate the builds. You don't need to duplicate maintenance work if you first extract a build template and simply keep the platform-exclusive parameters. You simply update the changes on the template and it will get automatically propagated to everything using it. It would also improve your ability to differentiate what is failing in case some build fails, you will know instantly whether it is on one platform or another.
Thank you for your insights. Indeed, lacking a better option I had the builds duplicated (though without using build templates). I will switch to build templates when the next maintenance is due. Still, I am a bit puzzled what's the worth of activating the dependency option "Run build on the SAME agent" when TeamCity nonetheless deems builds from DIFFERENT agents as suitable?! Anyway, it's good to know my chosen solution seems to be the correct one.
Run build on same agent is mainly focused on allowing two different builds the reuse of some shared resources, or intermediate steps which don't need to be published as artifacts. Most of the time this are simply multiple steps within the same build configuration, but sometimes they are better served separately and the "run on same agent" allows for that scenario.
The definition of suitable builds is very specific, and described here: https://confluence.jetbrains.com/display/TCD18/Snapshot+Dependencies