We are evaluating to migrate our C++ Win/OSX application CI from jenkins to TeamCity.
One open issue: how to best distribute our massive test-suite (single machine run-time can easily be >8h!) over as many build agents as we want to.
jenkins has a feature called "matrix builds" that makes this easy. Essentially, it allows you to create n variables with k_n discreet values, and will then trigger the resulting k_1 * k_2 * .. k_n build jobs.
For example, variables could be build_style (Release, Debug), platform (Mac, Win), wordsize (32, 64) - and the resulting 8 build-jobs would all be triggered from one build-configuration.
I already know TC can't do this. So I currently see the following options:
- create the build-confs by hand.
a) we are talking a lot here. 204 jobs, to be precise.
b) it's unflexible, if we want to re-distribute tests to configurations
c) I'm not sure, but I assume the display of this inside the TC gui isn't fabulous. But I might be wrong!
- create the whole bunch of build-confs through the REST-API. That kind of solves a) & b), but still leaves c)
- use HTTP to trigger custom builds, as described in http://confluence.jetbrains.com/display/TCD7/Accessing+Server+by+HTTP.
this would be my preferred solution. But it has a massive problem: how to wait for the completion of *all* 204 jobs to report a single outcome for the overall build?
We could dedicate a build-agent to wait for 204 artifacts to appear, and gather the results from that. But this would either waste resources in terms of CPU for those simple waiter-jobs, or we have to double our agent-licenses. Which means buying 40 instead of 20 or so. Which probably won't happen.
- forego TC, and use some home-grown system to dispatch & run the tests. Which somewhat defies the purpose of using TC to begin with ;)
So my question is: what would YOU do?
Thanks for your consideration,