Can you make TeamCity automatically re-run individual tests when they have failed

I have a set of tests written in C# that are run in TeamCity using the MSBuild runner type, and what i want to do is have team city automatically re-run any tests that fail, either straight away after being run, or at the end of the suite.

For example, if a test project gets a result of 2 Failed, 37 Passed, 2 Ignored, then i would want those 2 that failed to be re-run, as some of the problems that cause a test to fail are intermittent, and so a second run through could change the result to a pass.

Hence the above test result, when the 2 failures are re-run, could then turn into a 39 Passed, 2 Ignored result.

I'm not sure if team city can do this, perhaps it needs to be done from within the tests themselves, although i saw a note in the release files for the latest team city build talking about re-running failed tests, but couldn't quite figure out what it meant or how to use it from that documentation.

Any ideas?

thanks,
Peter

5 comments
Comment actions Permalink

ok, so after looking around for a while on the internet, i found that the solution was not to make team city re-run the test, but to do so within the test itself in c#. Which means this isn't really a question for team city, but i figured i'd place the answer i came up with here anyway for the sake of anyone who might come across this, and hopefully save them the hours i spent looking around.

There is an attribute you can assign to a test called [RepeatAttribute(numOfIterations)] which almost does what i wanted. It would run the test X times, but would stop if a test failed.

So, to change that to what i wanted, a repeat only on failure, up to X times, that would stop when it got a pass, i had to create my own attribute class that would work almost the same way as the repeat class, but change the exit condition to match my requirements.

You can find the initial code for the RepeatAttribute code here: http://code.google.com/p/mb-unit/source/browse/trunk/v3/src/MbUnit/MbUnit/Framework/RepeatAttribute.cs

And here is my edited version (I'll admit i don't quite know what every part is doing, but seemed to have figured it out enough to make it do what i want.)

namespace Tests.Common
{
    /// <summary>
    /// Decorates a test method and causes it to be re-run following failure until we get a pass.
    /// </summary>
    /// <remarks>
    /// <para>
    /// Each repetition of the test method will occur within its own individually labeled
    /// test step so that it can be identified in the test report.
    /// </para>
    /// <para>
    /// The initialize, setup, teardown and dispose methods will are invoked around each
    /// repetition of the test.
    /// </para>
    /// </remarks>
    /// <seealso cref="RepeatAttribute"/>
    [AttributeUsage(PatternAttributeTargets.Test, AllowMultiple = true, Inherited = true)]
    public class RepeatOnFailureAttribute :TestDecoratorPatternAttribute
    {
        private readonly int _maxNumberOfAttempts;

        /// <summary>
        /// Will re-run the test method each time we get a failure for a limited number of attempts.
        /// </summary>
        /// <example>
        /// <code><![CDATA[
        /// [Test]
        /// [RepeatOnFailure(3)]
        /// public void Test()
        /// {
        ///     // This test will be executed until we get a pass or have run it 3 times.
        ///     // Eg, if the first test run fails, we will run it again, and if the second attempt passes, then we will stop.
        ///     // if 3 attempts all fail, we dont try anymore
        /// }
        /// ]]></code>
        /// </example>
        /// <param name="maxNumberOfAttempts">The number of times to repeat the test while searching for a pass</param>
        /// <exception cref="ArgumentOutOfRangeException">Thrown if <paramref name="maxNumberOfAttempts"/>
        /// is less than 1.</exception>
        public RepeatOnFailureAttribute(int maxNumberOfAttempts)
        {
            if (maxNumberOfAttempts < 1)
                throw new ArgumentOutOfRangeException("maxNumberOfAttempts", @"The maximum number of attempts must be at least 1.");

            _maxNumberOfAttempts = maxNumberOfAttempts;
        }

        /// <inheritdoc />
        protected override void DecorateTest(IPatternScope scope, ICodeElementInfo codeElement)
        {
            scope.TestBuilder.TestInstanceActions.RunTestInstanceBodyChain.Around(delegate(PatternTestInstanceState state, Gallio.Common.Func<PatternTestInstanceState, TestOutcome> inner)
            {
                TestOutcome outcome = TestOutcome.Passed;
                int failureCount = 0;
                // we will try up to 'max' times to get a pass, if we do, then break out and don't run the test anymore
                for (int i = 0; i < _maxNumberOfAttempts; i++)
                {
                    string name = String.Format("Repetition #{0}", i + 1);
                    TestContext context = TestStep.RunStep(name, delegate
                    {
                        TestOutcome innerOutcome = inner(state);
                        // if we get a fail, and we have used up the number of attempts allowed to get a pass, throw an error
                        if (innerOutcome.Status != TestStatus.Passed)
                        {
                            throw new SilentTestException(innerOutcome);
                        }
                    }, null, false, codeElement);

                    outcome = context.Outcome;
                    // escape the loop if the test has passed, otherwise increment the failure count
                    if (context.Outcome.Status == TestStatus.Passed)
                        break;
                    failureCount++;
                }

                TestLog.WriteLine(String.Format(
                        failureCount == _maxNumberOfAttempts
                            ? "Tried {0} times to get a pass test result but didn't get it"
                            : "The test passed on attempt {1} out of {0}", _maxNumberOfAttempts, failureCount + 1));

                return outcome;
            });
        }
    }
}



So, you insert this file somewhere in your solution, then you can add it as an attribute above your test, eg:

[Test, RepeatOnFailure(2)]
public void SampleTest()
{
     // do something that will fail the first time the test is run
    // and then it will be run again, and this test will pass
    // or if it passes first time, it isn't run again
}

team city will then show you the latest result, so if it fails on the first X times, but passes on the next, you get a pass. And you can view the artifacts for that test run to see the expanded results of each test attempt.

0
Comment actions Permalink

Thank you for sharing the solution!

Our test reordering feature is quite different - it allows to run broken tests first, and get feedback faster.

Michael

0
Comment actions Permalink

Hi Peter,

I'm trying to create a similar repeat on failure in Nunit, but is is not working i.e test don't repeat upon failure. Can you look into and see what's incorrect in my code.



using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Reflection;
using System.Text;
using NUnit.Core;
using NUnit.Core.Extensibility;
using NUnit.Framework;
using NUnit.Framework.Extensions;

namespace CommonLibraries
{
    /// <summary>
    /// Decorates a test method and causes it to be re-run following failure until we get a pass.
    /// </summary>
    /// <remarks>
    /// <para>
    /// Each repetition of the test method will occur within its own individually labeled
    /// test step so that it can be identified in the test report.
    /// </para>
    /// <para>
    /// The initialize, setup, teardown and dispose methods will are invoked around each
    /// repetition of the test.
    /// </para>
    /// </remarks>
    /// <seealso cref="RepeatOnTestFailureAttribute"/>
    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
    public class RepeatOnFailureAttribute : Attribute
    {

        public int Num { get; set; }

        ///// <summary>
        ///// Will re-run the test method each time we get a failure for a limited number of attempts.
        ///// </summary>
        ///// <example>
        ///// <code><![CDATA[
        ///// [Test]
        ///// [RepeatOnFailure(3)]
        ///// public void Test()
        ///// {
        /////     // This test will be executed until we get a pass or have run it 3 times.
        /////     // Eg, if the first test run fails, we will run it again, and if the second attempt passes, then we will stop.
        /////     // if 3 attempts all fail, we dont try anymore
        ///// }
        ///// ]]></code>
        ///// </example>
        ///// <param name="maxNumberOfAttempts">The number of times to repeat the test while searching for a pass</param>
        ///// <exception cref="ArgumentOutOfRangeException">Thrown if <paramref name="maxNumberOfAttempts"/>
        ///// is less than 1.</exception>
        //public RepeatOnFailureAttribute(int maxNumberOfAttempts)
        //{
        //    maxNumberOfAttempts = Num;
        //}

        public int NumRepeated
        {
            get { return Num; }
        }
    }

    [NUnitAddin(Description = "Repeats the test on failure")]
    public class RepeatOnFailureTestDecorator : ITestDecorator, IAddin
    {
        public static string RepeatTestOnFailureAttributeTypeFullName = typeof(RepeatOnFailureAttribute).FullName;

        public bool Install(IExtensionHost host)
        {
            IExtensionPoint decorators = host.GetExtensionPoint("TestDecorators");
            if (decorators == null) return false;
            decorators.Install(this);
            return true;
        }
        public Test Decorate(Test test, MemberInfo member)
        {

            NUnitTestMethod testMethod = test as NUnitTestMethod;
            if (testMethod != null && testMethod.RunState == RunState.Runnable)
            {
                Attribute[] repearAttr = Reflect.GetAttributes(member, RepeatTestOnFailureAttributeTypeFullName, true);
                if (repearAttr != null)
                {
                    foreach (Attribute attr in repearAttr)
                    {
                        RepeatOnFailureAttribute repeatOnFailureAttr = attr as RepeatOnFailureAttribute;
                        if (repeatOnFailureAttr != null)
                        {
                            Console.WriteLine(string.Format("The Attribute available for test name {0}.The test name length {1}.", test.TestName.FullName, test.TestName.FullName.Length));
                            var testFullName = test.TestName.FullName;
                            test = new RepeatOnFailure(((NUnitTestMethod)test).Method, repeatOnFailureAttr.NumRepeated);
                            test.TestName.FullName = testFullName;

                        }
                    }
                }
            }
            return test;
        }



    }

    public class RepeatOnFailure : NUnitTestMethod
    {
        private MethodInfo _methodinfo;
        private int _testRunLoop;
        public RepeatOnFailure(MethodInfo method, int testRunLoop)
            : base(method)
        {
            Console.WriteLine("this is Nunit test method");
            _methodinfo = method;
            _testRunLoop = testRunLoop;
        }

        public override TestResult Run(EventListener listener, ITestFilter filter)
        {

            Console.WriteLine(string.Format("The reRun Alogorithm started for test {0}.{1}", _methodinfo.DeclaringType.FullName, _methodinfo.Name));
            var result = base.Run(listener, filter);
            if (result.IsFailure)
            {
                int i = 0;
                //When no result set previous result is still remembered.
                result.SetResult(ResultState.Inconclusive, string.Format("The test failed in the previous run. Rerun ({0}) started", (i + 1)), string.Empty);
                for (i = 0; i < _testRunLoop; i++)
                {
                    result = base.Run(listener, filter);
                    if (!result.IsFailure)
                        break;
                }
            }
            return result;

        }
    }

    public class sampletest
    {
        [Test, RepeatOnFailure(3)]
        public void SampleTest()
        {
            Console.WriteLine("hello");
            var a = 5;
            var b = 6;
            Assert.AreEqual(a, b);
        }



}

0
Comment actions Permalink

Hi Raj,

As i stated in my answer, i didn't fully understand all of the code i posted, just enough of it to make it work for what i wanted.

Briefly looking at your code, first thing i notice is that you have a lot more code in there than what I did which may be the issue? If writing in Nunit, approach i would use is to start by finding the code for the RepeatAttribute in nUnit, and just directly edit that code (same way i have done here).

I'll also mention that for me, the code only repeats on failure when i run it from team city, it doesn't do that when i run it locally ... is that the problem you are seeing?

Cheers

0
Comment actions Permalink

Hi Peter,

I used the same code as yours. Few queries:

1. What is PatternAttributeTargets.Test ? Its not identifying this.

2. IPatternScope scope, IcodeElementInfo are also not identified. Any library or nuget package required for this ?

 

Thanks

0

Please sign in to leave a comment.