Using VMs to Test Environment Permutations

We are developing a suite of enterprise products that are deployed into user environments with a variety of different configurations (e.g. different versions of Windows, Office, Sql Server, etc.)  We have an automated test suite that we want to run across all of the relevant permutations on each checkin.  We have 3 very powerful servers for virtualization that together can easily run 40+ of these configuration permutations.

TeamCity has the perfect mechanism for managing the distribution of tests to the appropriate VMs.  We could have a build runner on each VM that had properties describing its environment.  Then on the server, we would have build configurations for each project that included a list of the permutations for that project and triggered a test run after the central build is complete.  Nice!

Here's the problem.  JetBrains charges $399 per build agent (after 3), even those these would just be thin VMs running tests.  In total, that would be close to $15,000 in licensing, which seems rather expensive to me.  Does JetBrains offer the concept of a TestAgent like Microsft TFS does?  Is there some way to architect this that is reasonable but works around the licensing issue.

I don't mind the enterprise license fee and perhaps a few agents, but $15,000 isn't going to work for us.  In that case, I'd probably reengineer things to use TFS which has more of a fixed cost pricing structure.

Thanks,

-Scott

P.S. Please feel free to contact me on email if this can be resolved with a custom licensing agreement.

3 comments
Comment actions Permalink

Scott,

Sorry, I am not an expert in TFS, can you please describe what do you mean by TestAgent available in TFS?

Do you need to run all the 40 agents at the same time?

0
Comment actions Permalink

It might be useful to take a look at the following links:

http://blogs.msdn.com/lab_management/archive/2009/05/18/vsts-2010-lab-management-basic-concepts.aspx

http://blogs.msdn.com/lab_management/archive/2009/07/08/test-applications-on-virtual-environments-and-file-rich-bugs.aspx

 

For this product, which is available today in Beta 2 with a GoLive license, users have the ability to configure “environments” which are collections of VMs that need to be tested together (e.g. client and server).  Separately, you define the build machine(s), which can be local or remote.  Once a given build is complete (using their “build agent”), you can separately have tests run on a variety of “test agents” which directly correspond to the environments created above.  I’m not sure if the agents have different underlying code.  I do know that build and test agents are free, so for a large number of test permutations, TFS rapidly becomes less expensive than TeamCity while also having more extensive management capabilities around VMs (and work items, source control, et al which are sold separately in the JetBrains stack).

 

In TeamCity parlance, you can simulate much of the above by installing a build runner on each VM and setting environment variables to declare the various configuration details (e.g. version of office installed).  Then you create test projects that depend on the build project.  Those test projects are constrained to only run on VMs with build agents that declare the correct variables.  It’s not as rich as the environment model but it should get the job done.

 

These test build runners don’t all need to run simultaneously, but they do need to be authorized and connected at the same time.  Also, it seems artificial to run these in serial.

 

I understand this is how the pricing model works, but I think there is a qualitative difference between build agents which take a consistent heavy load and test agents which manage maybe a tiny 512MB WinXP VM and only execute 1 out of 40 or 50 total configurations.

 

Thanks,

-Scott

0
Comment actions Permalink

Scott,

Thank you for the information. Seems, guys in TFS Lab Management team do implement some interesting virtualization features.

We will probably compare our licensing model to theirs but this is not likely to turn into any immediate results.

Right now I know only one way to use many TeamCity agents for free except for evaluation: use EAP TeamCity versions (and bundled EAP license). Of course this is probably not what you want.

As a non-sales solution I can think only on ability to restrict the number of simultaneously connected agents instead of total number of authorized agents. This can probably be implemented as a plugin.

0

Please sign in to leave a comment.