Teamcity build execution with different combinations;

Hi,

        I am using Msbuild as a buildrunner to build my C# solution files and execution of test cases. I have configured 2 projects in the Teamcity, one project will build the solution file and the second one will executes the Nunit test cases. I have used the Build Dependency option among these 2 projects. Now, I got a requirement to execute the test cases with different session modes (ANSI, XYZ) as well as different session character sets (ASCII, UTF8, UTF16). Now I have to execute test cases against all the possible combinations of the each session mode and session character sets like below. I am using Nunit 2.4.8 addins for execution of the test cases.

ANSI ASCII

ANSI UTF8

ANSI UTF16

XYZ ASCII

XYZ UTF8

XYZ UTF16

In the above scenario, I have to execute the test cases 6 times. I have implemented custom tasks to accomplish this and I am able to execute the test cases. But, I don’t know how to identify particular test case failed against which combination.

Could you please let me know, How it could be done with simplified manner or  it could be done with other alternative.

Thanks in Advance

Prasad



4 comments
Comment actions Permalink

Could you please consider NUnit Row tests for that. Please have a look to blog post on that at http://mjeaton.net/blog/archive/2008/11/03/getting-started-with-nunits-rowtest.aspx or to nunit page for that at http://www.nunit.org/index.php?p=rowtest

There are more flexible features in NUnit 2.5 or MBUnit.

Thanks!

0
Comment actions Permalink

Hi ,

     Thanks for your reply. I have read the link that you provided for the Nunit row testing.If we enable this, we have to execute each test on all the possible combinations. Suppose, if we want to execute the tests manually for the single combination is not possible and every time we have to execute all the tests with all possible combinations.

I have implemented the custom tasks for achieving this. I have created 2 property variable for the session mode and session character. I am passing the values like (ANSI;XYZ & ASCII;UTF8;UTF16) to msbuild and i am taking the advantage of the cross-product. This cross-product is like foreach statement in C# like the below.

foreach (sessionmode smode in SM)
{
  foreaach (sessioncharacter sch in SC)
   {

      invoke nunit-console.exe

   }
}

I have implemented the above approach in the msbuild. Here, if i pass the sinlge combination also works fine. But, I am executing all the tests for given combinations, but i am unable to findout the test case failures (i.e  We dont know, the test case failed in which combination ). Teamcity just listing all the failed test cases and showing the stack-trace information.

Please let me know your suggestions and comments.

Thanks in Advance
Prasad

0
Comment actions Permalink

Hello,

You may compile tests assemblies several tymes for every combination of you SC and SM. So the pseudo-code would look like this:


foreach (sessionmode smode in SM)
{
  foreaach (sessioncharacter sch in SC)
   {

      var assemblies = compile_tests_assembly(sessionmode, sessioncharacter);

      invoke nunit-console.exe assemblies
   }
}

The idea here is to rename class FQ name by changing assembly name of the tests. This would make TeamCity to show thoses tests in different locales.

On the other hand, you may simply add a code that prints to console current test run mode for every test.

Thanks!

0
Comment actions Permalink

Eugene,

              Thank you very much. I'll will try and let you know the details.

Thanks

0

Please sign in to leave a comment.