I want to execute n test-runs against n tenants without having one build-config or build-step per tenant - how to best achieve?

We're using the rake-runner to execute cucumber against a set of Gherkin feature files.  

We have a platform that has n tenants (one per country that we operate in) where each tenant is an isolated deployment of the components making up our platform.

We need to run cucumber several times for different permutations of parameters (eg, browser user-agent) within one tenant, multipled by the number of tenants that we have.

We currently follow a branch-per-project model of development, and each branch gets its own TeamCity build-project.  At the moment, each build-project consists of 4 build-configs - Continuous, Duplicates, FunctionalTests, and ToQA.  The first three are triggered per-commit; the final one is a manual button-click.

FunctionalTests and ToQA are currently at 6 build-steps and counting.

We're not parallelising functional-test execution at the moment - this is an early-stage, and we haven't looked at it yet.  We may need to refactor to put the "execute functional tests" into a set of build-configs that are fan-out'd after Continuous finishes.

Our questions are this:

  • how would you arrange things in our scenario so that we minimise the amount of TeamCity admin/config we need to do when we create a branch-build?
  • what do we need to keep in mind if we want to parallelise the work in future?  Is it likely to be better to do this at the TeamCity level, or at the rake multi_task level?
  • is it possible to get TeamCity's rake-runner to keep executing even if a task fails (we'd like to run all permutations for a full report, rather than necessarily fail the whole build-step if one test fails)
  • what are our options for aggregating results together into a format that lends itself to at-a-glance consumption - eg, grid of ticks/crosses

I'm certain I haven't given enough information to answer the questions, but hopefully it's an interesting enough set of problems that we'll be able to get started down the right path...

Please sign in to leave a comment.