how to distribute test suite over many agents?

Hi there,

We are evaluating to migrate our C++ Win/OSX application CI from jenkins to TeamCity.

One open issue: how to best distribute our massive test-suite (single machine run-time can easily be >8h!) over as many build agents as we want to.

jenkins has a feature called "matrix builds" that makes this easy. Essentially, it allows you to create n variables with k_n discreet values, and will then trigger the resulting k_1 * k_2 * .. k_n build jobs.

For example, variables could be build_style (Release, Debug), platform (Mac, Win), wordsize (32, 64) - and the resulting 8 build-jobs would all be triggered from one build-configuration.

I already know TC can't do this. So I currently see the following options:

- create the build-confs by hand.

  a) we are talking a lot here. 204 jobs, to be precise.
  b) it's unflexible, if we want to re-distribute tests to configurations
  c) I'm not sure, but I assume the display of this inside the TC gui isn't fabulous. But I might be wrong!

- create the whole bunch of build-confs through the REST-API. That kind of solves a) & b), but still leaves c)

- use HTTP to trigger custom builds, as described in http://confluence.jetbrains.com/display/TCD7/Accessing+Server+by+HTTP.
   this would be my preferred solution. But it has a massive problem: how to wait for the completion of *all* 204 jobs to report a single outcome for the overall build?
   We could dedicate a build-agent to wait for 204 artifacts to appear, and gather the results from that. But this would either waste resources in terms of CPU for those simple waiter-jobs, or we have to double our agent-licenses. Which means buying 40 instead of 20 or so. Which probably won't happen.

- forego TC, and use some home-grown system to dispatch & run the tests. Which somewhat defies the purpose of using TC to begin with ;)

So my question is: what would YOU do?

Thanks for your consideration,

Diez