We are developing a suite of enterprise products that are deployed into user environments with a variety of different configurations (e.g. different versions of Windows, Office, Sql Server, etc.) We have an automated test suite that we want to run across all of the relevant permutations on each checkin. We have 3 very powerful servers for virtualization that together can easily run 40+ of these configuration permutations.
TeamCity has the perfect mechanism for managing the distribution of tests to the appropriate VMs. We could have a build runner on each VM that had properties describing its environment. Then on the server, we would have build configurations for each project that included a list of the permutations for that project and triggered a test run after the central build is complete. Nice!
Here's the problem. JetBrains charges $399 per build agent (after 3), even those these would just be thin VMs running tests. In total, that would be close to $15,000 in licensing, which seems rather expensive to me. Does JetBrains offer the concept of a TestAgent like Microsft TFS does? Is there some way to architect this that is reasonable but works around the licensing issue.
I don't mind the enterprise license fee and perhaps a few agents, but $15,000 isn't going to work for us. In that case, I'd probably reengineer things to use TFS which has more of a fixed cost pricing structure.
P.S. Please feel free to contact me on email if this can be resolved with a custom licensing agreement.