TeamCity 4.0 + MSBuild + NUnit + PartCover ???

TeamCity Gurus,

I am trying to figure out how to get TeamCity 4.0 (Build 7888) to display NUnit test results when ran from PartCover inside an MSBuild script. There has GOT to be someone out there that either 1.) wants the same thing or 2.) has already achieved this. I mean, come on… it is almost the year 2009. :) Here is some background information…

I have read numerous posts on how to get TeamCity to run NUnit using MSBuild so that the NUnit results are displayed in TeamCity. This doesn’t seem to be a problem as you can run NUnit with the TeamCity NUnit Test Launcher as described in this link in the documentation:

- http://www.jetbrains.net/confluence/display/TCD4/NUnitforMSBuild

This link essentially tells you to define the NUnit task to be the TeamCity NUnitLauncher MSBuild Task. That is great, but what if you want to generate coverage reports? I have found a few posts on how to do this with NCover. They are as follows:

- http://weblogs.asp.net/lkempe/archive/2008/04/29/run-ncover-through-msbuild-in-team-city.aspx
- http://weblogs.asp.net/lkempe/archive/2008/03/30/integration-of-ncover-into-team-city-for-tech-head-brothers.aspx

The problem with these Tech Brother’s posts (though they are VERY well written) is that the unit tests are run twice, once so TeamCity can capture the test history and a second so NCover can capture the code coverage. This is not ideal and is mentioned in the following posts that talk about this problem and present a better solution:

- http://getonthetrail.blogsite.org/Blog/post/2008/10/TeamCity-3x-NUnit-Under-NCover-21-Loss-of-Test-History.aspx
- http://getonthetrail.blogsite.org/Blog/post/2008/10/Leverage-NCover-in-Teamcity-without-loosing-Test-History.aspx

which reference this post where the solution originally came from:

- http://www.intellij.net/forums/thread.jspa?threadID=277581&tstart=0

These posts are GREAT as long as you want to use NCover. This solution essentially tells NCover to profile TeamCity’s runner process, not the NUnitLauncher process.

So let’s recap… with the last links, you can get TeamCity + MSBuild + NUnit + NCover to work. However, what about PartCover? LOTS of people use PartCover. I don’t believe that PartCover has the ability to attach to a running process like NCover does (with its use of the profileProcessModule attribute). So how would this work? Yes, you could pass the TeamCity NUnitLauncher exe to PartCover (just like they did with NCover), but this wouldn’t do any good since PartCover can’t attach to a different spawned process like NCover does. This would prevent the above solution from working using PartCover instead of NCover.

So… what about using the TeamCity NUnit addin? This REALLY sounds promising… Perhaps this is the key to getting TeamCity + MSBuild + NUnit + PartCover to work. Let’s check some documentation…

- http://www.jetbrains.net/confluence/display/TCD4/TeamCityAddinfor+NUnit

This says the following :

“TeamCity Addin for NUnit allows you to enable on-the-fly tests reporting from NUnit tests runner. TeamCity provides this addin for NUnit 2.4.6 and NUnit 2.4.7.”

This is EXACTLY what I want since I am going to be calling the original NUnit test runner with PartCover.exe. Apparently all I have to do is copy the following files from “C:\TeamCity\webapps\ROOT\update\plugins\dotnetPlugin.zip\dotnetPlugin/bin” to my Unit addins folder.

- JetBrains.TeamCity.NUnitAddin-NUnit-2.4.8.dll (2.4.7 is out there too and I tried that as well)
- JetBrains.TeamCity.NUnitAddin-NUnit-2.4.8-pdb (2.4.7 is out there too and I tried that as well)

I did this exactly as stated on the documentation link, but it didn’t work. These files had some dependencies that were not loading up. I also had to copy these files to the NUnit addins folder to get the TeamCity NUnitAddin to actually load up in NUnit. In fact if you use Reflector on JetBrains.TeamCity.NUnitAddin-NUnit-2.4.8.dll, it will have references to the below files. (UPDATE: I have since removed these files from the NUnit addins folder and the TeamCity NUnit Addin still loads up… so I am REALLY confused.)

- JetBrains.Container.dll
- JetBrains.Container.pdb
- JetBrains.TeamCity.NUnitCommon.dll
- JetBrains.TeamCity.NUnitCommon.pdb
- JetBrains.TeamCity.Utils.dll
- JetBrains.TeamCity.Utils.pdb

So, now that all 4 .dlls were in the NUnit addins folder, when NUnit ran, the TeamCity NUnit Addin actually loaded up which is shown in the NUnit log file “C:\NUnitPrimaryTrace.txt” with the below entries:

Loaded JetBrains.TeamCity.NUnitAddin-NUnit-2.4.8.dll
Registered addin: Teamcity Listener

I am taking this to mean that when the original NUnit runner is called, the TeamCity NUnitAddin is actually loading up inside of NUnit. So, given that, you would expect the tests results tab to show up in TeamCity, right? Wrong. At least not for me due to whatever I am doing wrong here. I can check the build.out folder of my build process and the tests ran and the coverage report was generated, but TeamCity didn’t have anything show up. It seems that the addin is loading up but TeamCity isn’t doing anything with it.

So to make a long story even longer, we are back to my original question… it is the question that drives us (at least me). How do you get TeamCity + MSBuild + NUnit + PartCover to actually work? It really seemed like the TeamCity NUnit addin was the answer, but it doesn’t seem to be working. I have also been reading up a bit about Gallio (http://www.gallio.org) and I believe that it has TeamCity integration and can also work with PartCover, but I have not gone down that path yet. Perhaps that is the solution I am looking for, but I sure thought that TeamCity NUnit addin was going to solve my problems. Am I not understanding its functionality properly? I assumed that it was the way to get TeamCity to listen if you are using the original NUnit runner (which is required if you are using PartCover).

I apologize if I have missed something or misunderstood something. I didn’t want to post without first trying everything I could think of as well as scouring the Internet for possible solutions, but it seems that no one has pulled off TeamCity + MSBuild + NUnit + PartCover yet. This makes me scratch my head. I can’t possibly be the only person that wants to figure this functionality out, but if someone has figured it out, either they didn’t post it or I didn’t find it.

Thanks for reading this, and feel free to blast me on any mistakes I have made. Any help is certainly appreciated.

Respectfully,

Shane

Oh... also, just for more enjoyable reading, here is a post to get NDepend running in TeamCity. I know that this post is not about that, but anyone reading this would probably be interested. As soon as I figure TeamCity + MSBuild + NUnit + PartCover out, I am going to add NDepend to the equation.

- http://weblogs.asp.net/lkempe/archive/2008/04/25/using-ndepend-in-team-city-build-management-tool.aspx

Edited by: staylor15 on Nov 7, 2008 1:10 AM

Edited by: staylor15 on Nov 7, 2008 1:21 AM

14 comments
Comment actions Permalink

Great text!

TeamCity Addin for NUnit should work without dependent libraries, because all necessary dependencies are embedded into that assembly. That is true Reflector will show those dependencies, but the very first action is to setup assembly resolver to resolve those dependencies.

I've checked PartCover. Nice tool.

TeamCity Addin for NUnit uses environment variables to get information about TeamCity build agent settings like logging port and build number.

Could you please check weather those environment variables exist in the environment of the NUnitLauncher.exe process that is started from the PartCover process. For that, please try executing 'cmd /c set' command instead of NUnitLauncher process.
Please attach the result here.

Could you please also check that the build agent service is not running under SYSTEM account.

Thanks!

0
Comment actions Permalink

Thanks for the info on the dependent libraries!

As for running "cmd /c set", remember, I am not spawning the NUnitLauncher.exe from inside the PartCover.exe (or at least I don't think I am). I am just calling the regular NUnit runner which is why the TeamCity Addin for NUnit really interested me. Perhaps you understood all this, so I am sorry if I am beating a dead horse. Also, my BuildAgent is not running as SYSTEM. I had originally created a TeamCity account (in the local admin group) that it is running under. At any rate, I modified my build script to call "cmd /c set" before my PartCover.exe call and I also passed it to the PartCover.exe call (which caused it to hang, but the log file still produced the necessary information). Here is what I get:

: =========================================================================
: ECHO ALL THE ENVIRONMENT VARIABLES THAT ARE SET
: cmd /c set
: ALLUSERSPROFILE=C:\Documents and Settings\All Users
: BUILD_NUMBER=38
: BUILD_NUMBER_FORMAT= : BUILD_VCS_NUMBER=3780 : BUILD_VCS_NUMBER.svn__https___source_acculence_com_Acculence_Core_trunk_packages_full=3780 : BUILD_VCS_NUMBER_1=3780 : ClusterLog=C:\WINDOWS\Cluster\cluster.log : CommonProgramFiles=C:\Program Files\Common Files : COMPUTERNAME=BUILD : ComSpec=C:\WINDOWS\system32\cmd.exe : FP_NO_HOST_CHECK=NO : NUMBER_OF_PROCESSORS=1 : OS=Windows_NT : Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem : PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH : PROCESSOR_ARCHITECTURE=x86 : PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel : PROCESSOR_LEVEL=6 : PROCESSOR_REVISION=0f08 : ProgramFiles=C:\Program Files : PROMPT=$P$G : SystemDrive=C: : SystemRoot=C:\WINDOWS : teamcity-dotnet-log-file=C:\BuildAgent\temp\buildTmp\TeamCity.NET\.net.2008-11-07T09-45-16.#2712-238225866114545.log : TeamCity.Agent.NET.root.cdc51fb1-1a81-4792-8831-3a5a69c3af76=C:\BuildAgent\plugins\dotnetPlugin\bin : TeamCity.AgentUrl.cdc51fb1-1a81-4792-8831-3a5a69c3af76=http://localhost:9090/RPC2 : teamcity.build.properties.file=C:\BuildAgent\temp\agentTmp\teamcity.build55694.properties : TeamCity.BuildId.cdc51fb1-1a81-4792-8831-3a5a69c3af76=39 : teamcity.dotnet.coveragelauncher=C:\BuildAgent\plugins\dotnetPlugin\bin\TeamCityRunners\JetBrains.dotTrace.TeamCity.CoverageRunner.exe : teamcity.dotnet.nunitaddin=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.TeamCity.NUnitAddin-NUnit : teamcity.dotnet.nunitlauncher=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe : teamcity.dotnet.nunitlauncher.msbuild.task=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.MSBuildLoggers.dll : teamcity.dotnet.nunitlauncher1.1=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher1.1.exe : teamcity.dotnet.nunitlauncher2.0=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher2.0.exe : teamcity.dotnet.nunitlauncher2.0.vsts=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher2.0.VSTS.exe : teamcity.dotnet.platform=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.TeamCity.PlatformProcessRunner.1.1.exe : TEAMCITY_BUILDCONF_NAME=Integration : TEAMCITY_JRE=C:\BuildAgent\jre : TEAMCITY_PROJECT_NAME=Acculence.Core : TEAMCITY_VERSION=4.0 EAP (build 7888) : TEMP=C:\BuildAgent\temp\buildTmp : TMP=C:\BuildAgent\temp\buildTmp : USERDNSDOMAIN=ACCULENCE.LOCAL : USERDOMAIN=ACCULENCE : USERNAME=teamcity : USERPROFILE=C:\Documents and Settings\teamcity : windir=C:\WINDOWS : WRAPPER_ARCH=x86 : WRAPPER_BITS=32 : WRAPPER_FILE_SEPARATOR=\ : WRAPPER_OS=windows : WRAPPER_PATH_SEPARATOR=; : ========================================================================= : PARTCOVER TO ECHO ALL THE ENVIRONMENT VARIABLES THAT ARE SET : regsvr32.exe -s "C:\BuildAgent\work\c9b30c5a3b3584c\src\..\build\tools\PartCover\PartCover.CorDriver.dll" : "C:\BuildAgent\work\c9b30c5a3b3584c\src\..\build\tools\PartCover\PartCover.exe" --target=c:\windows\system32\cmd.exe --target-args="/c set" : ALLUSERSPROFILE=C:\Documents and Settings\All Users : BUILD_NUMBER=38 : BUILD_NUMBER_FORMAT=
: BUILD_VCS_NUMBER=3780
: BUILD_VCS_NUMBER.svn__https___source_acculence_com_Acculence_Core_trunk_packages_full=3780
: BUILD_VCS_NUMBER_1=3780
: ClusterLog=C:\WINDOWS\Cluster\cluster.log
: CommonProgramFiles=C:\Program Files\Common Files
: COMPUTERNAME=BUILD
: ComSpec=C:\WINDOWS\system32\cmd.exe
: Cor_Enable_Profiling=1
: Cor_Profiler={717FF691-2ADF-4ac0-985F-1DD3C42FDF90}
: FP_NO_HOST_CHECK=NO
: NUMBER_OF_PROCESSORS=1
: OS=Windows_NT
: PartProfile_Message_Pipe=
.\pipe\PartCover.MessageCenter.2964.584315781
: Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem
: PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
: PROCESSOR_ARCHITECTURE=x86
: PROCESSOR_IDENTIFIER=x86 Family 6 Model 15 Stepping 8, GenuineIntel
: PROCESSOR_LEVEL=6
: PROCESSOR_REVISION=0f08
: ProgramFiles=C:\Program Files
: PROMPT=$P$G
: SystemDrive=C:
: SystemRoot=C:\WINDOWS
: teamcity-dotnet-log-file=C:\BuildAgent\temp\buildTmp\TeamCity.NET\.net.2008-11-07T09-45-16.#2712-238225866114545.log
: TeamCity.Agent.NET.root.cdc51fb1-1a81-4792-8831-3a5a69c3af76=C:\BuildAgent\plugins\dotnetPlugin\bin
: TeamCity.AgentUrl.cdc51fb1-1a81-4792-8831-3a5a69c3af76=http://localhost:9090/RPC2
: teamcity.build.properties.file=C:\BuildAgent\temp\agentTmp\teamcity.build55694.properties
: TeamCity.BuildId.cdc51fb1-1a81-4792-8831-3a5a69c3af76=39
: teamcity.dotnet.coveragelauncher=C:\BuildAgent\plugins\dotnetPlugin\bin\TeamCityRunners\JetBrains.dotTrace.TeamCity.CoverageRunner.exe
: teamcity.dotnet.nunitaddin=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.TeamCity.NUnitAddin-NUnit
: teamcity.dotnet.nunitlauncher=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe
: teamcity.dotnet.nunitlauncher.msbuild.task=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.MSBuildLoggers.dll
: teamcity.dotnet.nunitlauncher1.1=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher1.1.exe
: teamcity.dotnet.nunitlauncher2.0=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher2.0.exe
: teamcity.dotnet.nunitlauncher2.0.vsts=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher2.0.VSTS.exe
: teamcity.dotnet.platform=C:\BuildAgent\plugins\dotnetPlugin\bin\JetBrains.TeamCity.PlatformProcessRunner.1.1.exe
: TEAMCITY_BUILDCONF_NAME=Integration
: TEAMCITY_JRE=C:\BuildAgent\jre
: TEAMCITY_PROJECT_NAME=Acculence.Core
: TEAMCITY_VERSION=4.0 EAP (build 7888)
: TEMP=C:\BuildAgent\temp\buildTmp
: TMP=C:\BuildAgent\temp\buildTmp
: USERDNSDOMAIN=ACCULENCE.LOCAL
: USERDOMAIN=ACCULENCE
: USERNAME=teamcity
: USERPROFILE=C:\Documents and Settings\teamcity
: windir=C:\WINDOWS
: WRAPPER_ARCH=x86
: WRAPPER_BITS=32
: WRAPPER_FILE_SEPARATOR=\
: WRAPPER_OS=windows
: WRAPPER_PATH_SEPARATOR=;

Edited by: staylor15 on Nov 7, 2008 7:53 PM

0
Comment actions Permalink

Does this help or do you need me to gather some more info? Thanks!

0
Comment actions Permalink

Was that done from the build you are trying to run with PartCover?
All necessary environment variables have been passed to the NUnitLauncher process.

What version of NUnit do you use? And what version of NUnit addin was that?

Could you please check you build scripts does not filter/alter environment for child processes.

Please note, all tests has to be done from build that is running on build agent.

Thanks, sorry for delay.

0
Comment actions Permalink

Eugene,

Thanks for the reply. The version of NUnit that I am using is 2.4.8 and the TeamCity NUnit Addin is also 2.4.8 (see the original post for the files that I have inside of the NUnit addins directory). This was done inside the build that is initiated from TeamCity and runs in the Build Agent (which is where I pulled the log files from). The MSBuild script doesn't seem to be changing anything in child processes. In fact, the second time the environment variables are output, the variables are from the PartCover.exe process since command is passed to PartCover. The first time is right before PartCover. So, before the PartCover call and during the call, the environment variables seem to be set properly.

Here is the actual MSBuild call:

]]>

It seems like everything is doing what it is supposed to be doing (including the loading of the NUnit TeamCity Addin), but when you log into TeamCity and look at the build, there are no tests that show up for the build. Sorry for being so difficult...

Edited by: staylor15 on Nov 10, 2008 11:16 PM

0
Comment actions Permalink

Ok. Could you please enable logging for the task. For that please create a file ''teamcity-log4net.xml'' with the following content:

		
	
		
                    
		
	

        
             
             
             
             
             
             
             
             
                  
             
        

		
           
 	   
           
	
    
]]>


This file should be placed near TeanCity Nunit Addin 2.4.8 dll.

Running with that option should produce console output and log file output for the process. Log files can be found in %TEMP%/TeamCity.NET folder. Please remove all files from the folder before run.
By default those files are save to ]]>/temp/buildTmp folder. Pleaes note, that folder will be cleaned on the next build start.

Please attach achieved logs to the issue.

BTW. Do you have some custom settings for .NET Framework?
BTW2. Is that possible to run the tests and get 'Tests' tab using the same NUnit.Console but without PartCover?

Thanks!

0
Comment actions Permalink

I am not aware of any custom settings for the .Net framework that are set.

I ran the build through TeamCity with PartCover calling the NUnit console (with the teamcity-log4net.xml sitting next to the TeamCity addin in the NUnit addins folder) and there were log files generated where you told me to look. I then did your suggestion of running the build with the NUnit console ONLY without being called through PartCover. The TeamCity NUnit addin seems to be loading as shown in NUnitPrimaryTrace.txt (attached) and there were log files generated (attached as well). This run takes PartCover out of the equation and there still isn't a test tab showing up in TeamCity for that build even though the tests clearly run as we get the test output in our build.out folder. The command that gets run (there are two assemblies being ran which is why there are two log files) is as follows:

- "C:\BuildAgent\work\c35ed316d4005f7c\src\..\build\tools\NUnit\nunit-console.exe" Acculence.Core.Tests.dll /nologo /xml="C:\BuildAgent\work\c35ed316d4005f7c\src\..\build.out\tests\Acculence.Core.Tests.xml" /output="C:\BuildAgent\work\c35ed316d4005f7c\src\..\build.out\tests\Acculence.Core.Tests.out" /err="C:\BuildAgent\work\c35ed316d4005f7c\src\..\build.out\tests\Acculence.Core.Tests.err"

I also tried this command removing the "xml" attribute and it still didn't work. I must be bad luck as I can't seem to get the TeamCity NUnit addin to work even when calling nunit-console.exe directly (without being called through PartCover). I have attached a pic of this too.

Let me know if there is anything else you would like me to try. Thanks!



Attachment(s):
TeamCity Build Results.JPG
NUnitPrimaryTrace.txt
.net.2008-11-11T11-06-30.#2600-241730611779295.log.debug.txt
.net.2008-11-11T11-06-28.#3216-241730590211551.log.debug.txt
0
Comment actions Permalink

I need to check that in more details. I'll reply you soon.

0
Comment actions Permalink

I've checked NUnit 2.4.8 runner and found out the it tries to load assembly according to created AssemblyName object. But, somehow 'addins' folder of NUnit 2.4.8 is not included in application probe paths. Thus NUnit 2.4.8 failes to load any addin in console mode. There are several workarounds for that:
- copy TeamCity addin assembly both to NUnit bin and bin/addins folders
- patch NUnit-Console.exe.config to include addins to application probe paths. Add this into config/runtime elenent

      
   
]]>

- use NUnit 2.4.7

I going to add the bug report to NUnit for 2.4.8 on that as well.

I hope that will resolve your issue.

0
Comment actions Permalink

Please don't hate me :)

I had made this fix a while ago when I read the following link:

- http://nunit.com/blogs/?p=56

I have attached the nunit-console.exe.config (as well as the nunit.exe.config for good measure) and you will see that the addins section is in both of them (as I did have to add it to nunit-console.exe.config).

Also, we know that the TeamCity NUnit addin is loading properly from the addins directory because it is showing up in the NUnitPrimaryTrace.txt (also attached again). If I remember correctly, I think it even loads if that section is NOT included in nunit-console.exe.config (weird). I could be wrong, but I believe it was loading before I ever added that section to that file.

Also, also, if the TeamCity NUnit addin wasn't loading, it wouldn't have produced the debug output when I turned the logging up for the addin. We placed the log4net config file in the addins directory along side the TeamCity NUnit addin. When we ran the build, it had to have got loaded because we got the debug output.

Please don't give up on me! :) You are my only hope on this!



Attachment(s):
NUnitPrimaryTrace.txt
nunit.exe.config
nunit-console.exe.config
0
Comment actions Permalink

Sorry. You were right to point me that TeamCity addin somehow was loaded partly.
What build of TeamCity was that?

Could you please try running some simple test to check the issue source.

Please check that NUnit.Core.Interfaces assembly reference in NUnit Addin for TeamCity are equal to
the same assembly in you NUnit distribution.

In
ftp://ftp.intellij.net/pub/.teamcity/dotNetListeners/dotnet-listeners-4.0.283.47914.zip
I have attached the latest build of DotNet part, could you please try running it with that one.
To patch .NET part of .NET runners please do the following:

1. Open /webapps/root/update/plugins/ 2. Copy dotNetPlugin.zip to temp folder 3. Back up dotNetPlugin.zip 4. Extract dotNetPlugin.zip 5. Replace contents of /bin folder with new files. 6. Pack files again. Make sure there is no files in the root of archive. 7. Replace dotNetPlugin.zip file on the server. 8. All build agents should upgrade automatically. 9. Run the builds. You may patch only one build agent. For that, please replace ]]>/plugins/dotnet-plugin/bin folder with contents of the .zip file.

Thanks!

0
Comment actions Permalink

That fixed it! Apparently there was something wrong with TeamCity Build 7888. You are the MAN!

Are you going to release another EAP build? Or is it in one already?

0
Comment actions Permalink

Is there anything else that I need to do before you put this in a new build? Thanks!

0
Comment actions Permalink

It is already in the trunk. Next EAP will contain that fix.

0

Please sign in to leave a comment.