UnitTests failing using NUnit GUI but passing with TeamCity's NUnit Runner

Hello,

I have some code with unit tests that pass in a Debug build but fail in a Release build.  On top of that, the tests only fail when using NUnits GUI but pass with TeamCitys NUnit runner even though they are both the same version.

Because of this we can't trust TeamCity until we've got it sorted.

I know this is not a .Net forum but here is the offending test code, just to give you in idea of why it's failing for a Release build - it's basically because of reduced stack information due to code optimization.


[TestFixture]
public class TestLogger
{
     [Test]
     public void GetCurrentClassLoggerReturnsLoggerWithOwningTypeName()
     {
          var logger = Logger.GetCurrentClassLogger();
          Assert.AreQueal(logger.Name, GetType().Name); }
     }

     public class Logger
     {
          public static ILogger GetCurrentClassLogger()
          {
               var declaringType = new StackFrame(1, false).GetMethod().DeclaringType;
               return new Logger(declaringType.Name, declaringType.FullName);
          }
     }


Any ideas ?

Thanks

9 comments
Comment actions Permalink

Hi Mick

What version do you use?
We fixed similar issue TW-17679 in 6.5.3.

0
Comment actions Permalink

Hi Michael

TeamCity version is 6.5.5 (build 18087)

NUnit-2.5.10 v4.0 MSIL

Will look at post, thanks

0
Comment actions Permalink

We'll continue troubleshooting in TW-19677

0
Comment actions Permalink

Consider to use [MethodImpl(MethodImplOptions.NoInlining)] for the method to avoid it possible inlining by the runtime.

0
Comment actions Permalink

Thanks for the reply.

That sounds great for solving that particular problem, but what about the ones in the future that I won't be able to spot because the tests pass when they should fail ?

0
Comment actions Permalink

What TeamCity does is to start a processs for your assemblies to run tests. There could be some .NET specific picularities that could affect tests behavoiur. Most likely, you need to have your own .NET Framework host to control it. Another trick is to try changing .NET Framework default settings for the machine to let .NET Framework bahave in expected way.

0
Comment actions Permalink

It's probably best if I re-clarify the problem;

Consider the following complete code example (slighly different to the previous sample) ;

using System.Diagnostics;
using NUnit.Framework;

namespace TeamCityTest
{
    [TestFixture]
    public class TestLogger
    {
        [Test]
        public void GetCurrentClassLoggerReturnsLoggerWithOwningTypeName()
        {
            Assert.AreEqual(Logger.GetCurrentClassLogger(), GetType().Name);
        }
    }

    public class Logger
    {
        public static string GetCurrentClassLogger()
        {
            return new StackFrame(1, false).GetMethod().DeclaringType.Name;
        }
    }
}

You should be able to build this code and see for yourself.  If this code is built using Visual Studio and the tests run, then it passes in Debug mode but fails for Release mode.  This is what we expect - at this stage we want this to happen !  However TeamCity is reporting that the same tests are passing for Debug *and* Release mode.  We have tried this on more than one machine and we get the same results.

0
Comment actions Permalink

Please try running compiled assemblies with NUnit without VS, i.e. from commandline.
What have you specified as build options when you build it under TeamCity?

I belive the result only depends on .NET runtime as mentioned method is always avaialble in the generated assemblies.

0
Comment actions Permalink

I've just tried running the command line as follows:

nunit-console TeamCityTest.dll

The results are still the same as in Visual Studio.  We are using the resharper test runner in visual studio anyway.

0

Please sign in to leave a comment.