MbUnit test results
I'm trying to get MbUnit to play nicely TeamCity 3 (so it will display Tests passed / failed and show statistics as it does with JUnit). My projects are all MSBuild projects and they current run MbUnit correctly and generate an html report that i can include as an artifact.
From reading around the docs, my plan was to generate a teamcity-info.xml with the MbUnit statistics (in the hope that TeamCity would automagically pick them up and process them).
I've got it generating a teamcity-info.xml that looks like
Tests passed: 2
]]>
Which seems to be automatically included as an artifact by TeamCity. However, there doesn't seem to be any processing beyond that. Am I approaching this in the correct way, and if so, what properties do I need to put in my teamcity-info file to get this to work?
Aside from this issue I'm very impressed - we've been using CCNET for the last couple of years and this has been far easier to get up and running and extend!
Many thanks,
Will Holley
Please sign in to leave a comment.
Will, I'm assuming you have checked out this page:
http://www.jetbrains.net/confluence/display/TCD3/IncludingThird-PartyReportsintheBuildResults
The part you are missing is at the very bottom where you need to add statisticValue elements inside the build element.
Tests passed: 2 * * You also need to edit the TeamCity data directory/config/main-config.xml file to include the new stats you want to track. So if you want to add the 2 keys above you would add: The ]]> element that you specified will just modify the build status text (usually just says "Successful", "Failed", etc).
Thanks Jeff,
I had looked at the page but couldn't work out whether there were some standard keys to make this work or whether i had to define custom keys. Anyway - all looks good now!
Will
I am now generating a teamcity-info.xml which is getting automatically picked up. This looks like:
Tests passed: 2 Tests failed: 0 I've also changed the main-config.xml so it now contains: ]]>
The graphs are now showing up in the statistics tab but they don't display any data! The status labelling doesn't appear to have any effect. Is there something else I need to do?
Hello Will,
In fact, there is a nasty bug with teamcity-info.xml processing, and this bug was fixed right after the last EAP was released. Your setup looks correct, but I'm afraid you need to wait for the next EAP build (which hopefully will be released on the next week).
Really sorry for that :(
Kind regards,
KIR
No problem - looking forward to the next code drop! :)
"Will Holley" <no_reply@jetbrains.com> schrieb im Newsbeitrag
news:27426074.1195203933511.JavaMail.itn@is.intellij.net...
Have you MbUnit Working?
May you share your Information ;)
Regards
Albert
http://der-albert.com
Hi Albert,
I have MbUnit working, yes. The simplest way to do it is just to ensure MbUnit runs as part of the build script (we use MsBuild to run our .csproj projects and have MbUnit run as part of that using the MbUnit MsBuild task) - this generates a html file which you can include in the build artifacts for you TeamCity build configuration.
If you want to go further than that, as we have, you could get your build script to produce a teamcity.info file including the number of tests passed, failed, etc. To get the numbers out of MbUnit requires a patch to the MsBuild task (which you can find on the MbUnit Jira site or I can email to you) and we've written a couple of MsBuild tasks to produce a teamcity.info file which we could also make public.
Once thats done, the test statistics can be viewed directly in the build history in TeamCity.
I think I'll put a blog post up about this as it's a little involved, but the end result is quite nice. I'll post here again when it's up.
"Will Holley" <no_reply@jetbrains.com> schrieb im Newsbeitrag
news:16819264.1197222419663.JavaMail.itn@is.intellij.net...
Mailing me would be great, making the whole thing also public will help much
more people :)
info@der-albert.com
Regards
Albert
Does your custom change show passed,failed,new counts under the projects tab?
I currently use NUNIT for some .NET testing. But I want to use a framework called zanebug and still have all the stats and numbers show up in TeamCity like NUNIT results do currently.
The output looks like this under the project tab for NUNIT.
Tests failed: 3 (3 new), passed: 6 No artifacts No changes 10 Dec 14:10 (1m:11s)
And this is how I want zanebug to look.
Thanks
Bob
Sorry I meant to reply to the parent node of this discussion.
Message was edited by:
BobMN
Message was edited by:
BobMN
Y I made an MsBuild task to generate the teamcity-info.xml which is parsed to display these stats - I'll put them on my blog soon but have been busy this week and haven't got around to it.
Once you have the teamcity-info.xml being generated in the build working directory, status messages get automatically added to the dashboard. You can also put stats in there which can be used in graphs by editing the server configuration files.
Can you please give a link to where you upload the file.
thanks
I actually just started working on a gallio teamcity testrunner. I was going to just do a mbunit runner but with gallio being able to run multiple testing frameworks and being able to do some cool reporting I figured I would target that. I am hoping to have something in the next couple weeks. I need to check about licensing, but it will probably get included as an opensource module inside of gallio.
It will give you rpc style reporting like the nunit teamcity testrunner, include standard team city test reporting and charts, as well as add a "Test Details" tab with whatever extra reporting you decide to configure from the gallio runner.
One problem I will have is using it in the build. I don't think I will have the luxury of patching the build files like current TC runner does for NUnit. Maybe I can work something in with the standard Gallio runners and if it detects its in a TC build it can just add my TC logger and progress monitor.
I will post here when I have something working.
Adam - it may be a dirty hack but you can look for the presence of the teamcity environment variables as an indication that you're running in TC. I'm working on putting a blog up post about MbUnit integration with TC as soon as I get the time and also adding the MSBuild tasks to output teamcity log files to the MSBuild Community Tasks project. If you want any help with Gallio to integration, drop me an email.
Will
I wouldn't call it dirty, that is what I was planning on using. If it detects them it will just append my tc rpc test logger and progress monitor.
>I'm working on putting a blog up post about MbUnit integration
I was going to do the same with ncover, I just want to figure out how to get better charts first.
Not sure what you mean about teamcity log files, currently tc wraps msbuild so all logging gets captured and sent back to the build server.
Thanks Will, I am take you up on that. I just want to spike something first and get into the gallio code base, then everyone can work on patches for it. I already talked to Jeff Brown and he wants to include as part of the Gallio base if licensing permits.
Adam,
Please do not hesitate to contact TeamCity Team if you have any questions while your work on the plugin.
You can send e-mails to teamcity-feedback@jetbrains.com
--
Best regards,
Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"
I've added the MSBuild tasks and a blog post at http://bluewireblog.wordpress.com/2008/01/24/teamcity-msbuild-tasks/ - let me know of any probs.
Will
I started the gallio test monitor, from my end its pretty complete I just can't get it to work with TC. I sent an email asking jetbrains for help.
I created both an rpc monitor and a console monitor for the new service messages in tc 3.1. I can't get service messages to work under msbuild and I have no idea where or why my rpc calls are failing. As far as I can tell its the exact same calls and xml as the jetbrains nunit runner.
Hopefully we will get something soon.
Adam
Bumping this... I sent an email last week to support asking for some help in tracing and debugging rpc calls to the TC build agent service.
Any word? Below is a copy of the email.
Hi Yegor,
Back in this thread I talked about building a plugin for unit testing
with mbunit and gallio.
http://www.intellij.net/forums/thread.jspa?forumID=68&threadID=271219
I have since spiked it but its not working. Can you provide some
assistance for debugging the rpc calls on the agent... maybe some log4j
settings to get more verbose output.
All of my tests pass. I verified to the best of my ability I am
generating the same xml as the jetbrains test runner. I am use the
same cookcomputing.xmprc.dll. But nothing seems to get captured and logged.
I also created a console logger to try and get it to work with the new
service messages 3.1 eap. I cannot get service messages to work
either via MsBuild. If I log straight to Console.Out it seems tc
doesn't pick them up, I don't see them in the build log on the server,
however if I run local with /v:diag I see the Console.Out messages.
So then I tried writing a message with the msbuild logger and the
output seems to get mangled because msbuild uses in its log
messages too.
Here is what shows up in the msbuild build file if I log using the
msbuild logger. The double hash gets dropped and for some reason the
test messages, not the suite messages, drop the closing ]
: [Target "Test" in project
"Build.msbuild.teamcity.patch.proj"] teamcity[testSuiteStarted
name='Gallio.TeamCity.Monitor.Sample.Tests.DLL']
: [Target "Test" in project
"Build.msbuild.teamcity.patch.proj"] teamcity[testFinished
name='One_Plus_One_Is_Two'
: [Target "Test" in project
"Build.msbuild.teamcity.patch.proj"] teamcity[testStarted
name='One_Plus_One_Is_Two'
: [Target "Test" in project
"Build.msbuild.teamcity.patch.proj"] teamcity[testSuiteFinished
name='Gallio.TeamCity.Monitor.Sample.Tests.DLL']
And via console.out which never shows up in the build log...
##teamcity[testStarted name='One_Plus_One_Is_Two']
Any help you could provide would be very beneficial, Gallio is a sweet
test running platform. NUnit, MbUnit, XUnit, Waitn with screen shot
capturing... I really want to get this out there I just seem like I
hit a dead end.
It will be oss so I have no problem sending you my code either.
Thanks,
Adam Tybor
Hello,
As I know one of our developers has sent you reply on 21 Feb. It seems you
did not received it. Message contained an attachment of size 2.7Mb, probably
attachment was too large and message was not delivered to you. I can try to
resend it to you again.
--
Pavel Sher
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"
"Adam Tybor" <adam.tybor@gmail.com> wrote in message
news:1410725.1204043445708.JavaMail.itn@is.intellij.net...
>
>
>
>
>
>
>
>
>
>
>
>
Ah... if it was sent to gmail it probably bounced if there were any binaries in it.
Can you try sending it to adam-tybor@@@raddevgrp-com
replace the hyphens with dots, and only one @ :)
I've forwarded email to the new address.
--
Pavel Sher
Software Developer
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"
"Adam Tybor" <adam.tybor@gmail.com> wrote in message
news:18052225.1204048258346.JavaMail.itn@is.intellij.net...
>
>
Did you try using the XSLT support on the MbUnit console?
http://docs.mbunit.com/help/html/gettingstarted/MbUnitConsoleRunner.htm
I was thinking of going this route instead, although I currently have the NANT custom task running MbUnit just fine.
Adam, you have mentioned you've been working on gallio test runner. Is that available somewhere? Do you have some problems doing it? Have you managed to get any positive result?
If you're using NAnt and MbUnit with TeamCity you may want to take a look at the NAnt-Extensions project.
It includes a NAnt task for MbUnit that integrates nicely with TeamCity.
The project also includes additional tasks that make it easy to interact with TeamCity from a NAnt build script.
Looks promising! Feel free contact JetBrains TeamCity team at teamcity-feedback[]jetbrains.com
MBUnit Gallio team has added support for TeamCity. It is available in Gallio trunk. To enable it one have to add bundled plugin to the task for NAnt, MSBuild or to Gallio.Echo