Is it possible to output HTML as part of ##teamcity[testFailed] tag?

I am still trying to integrate fitnesse even more tightly. One missing feature we really want is to be able to display the fitnesse test output relevant to a failed test right as it fails (very much like junit plugin does).
Is it possible to output HTML formatted text in the message attribute of a testFailed tag?
The entire test could be multiple page long. In theory I might be able to only output the relevant table with the failure. In any case it would be multi-line output. It would be nice to have a escape tag set like the multi-line of groovy/ruby that would allow to output anything between these tags. I did not see this in the docs so I will assume it isn't available. How do you suggest I do this (without a full custom plugin obviously)?

Thanks in advance.

Jacques

10 comments
Comment actions Permalink

In TeamCity 4.0 we've added ability to plug an extension to the server to format stacktrace for showing on the web (when you click "Show details" link). The extension is called jetbrains.buildServer.serverSide.FailedTestOutputFormatter. So you need to implement this interface and either register it as extension in jetbrains.buildServer.serverSide.ServerExtensionHolder or just specify implementing class it in the Spring configuration file.

--
Pavel Sher

0
Comment actions Permalink

Thanks for the quick answer.
I have more questions now :-):
1. How would I know that a test came from a specific step of the build?
I do not want to try to guess from the name of the test since this could be error prone or ugly.
Right now some build use ant and some are using maven. In ant, there is a top level target to run unit tests and one for fitnesse test. Could I get to the current task or the chain of tasks that trigger that test?
With maven we have a custom ant runner that does the same. I am not sure how the build tasks would look like then.

2. The javadoc on FailedTestOutputFormatter says that there can be only one. I suppose this means I have to wrap the default one to get the built-in behavior. So how do I get the default one?
From an API this may not be optimal. I believe it would be too limiting as different test frameworks used in different builds (.NET, web java app, desktop app...) would probably use different functional test frameworks and therefore would all need different custom output. A filter chain may be more appropriate with the ability for one formatter to stop the chain once it has rendered the output. My first issue will have to be addressed in a deterministic manner also.

Jacques

Edited by: Jacques Morel on Nov 20, 2008 9:53 AM

0
Comment actions Permalink

1. How would I know that a test came from a specific step of the build?


Not sure I understand you. You probably could write some tag to stacktrace to indicate that it need to be formatted.

2. The javadoc on FailedTestOutputFormatter says that there can be only one. I suppose this means I have to wrap the default one to get the built-in behavior.


In the new EAP build (which we hope to release this week, probably today) there will not be such restriction. There will be method canFormat(SBuild, FailedTestOutputBean) and if it returns true then this formatter will be used. If none of the formatters returns true then default formatter is used which simply shows stacktrace as is.

The new interface:

/**

  • Extension point for providing new logics for test's stacktrace formatting.

*/
public interface FailedTestOutputFormatter extends ServerExtension {

/**

  • Formats test output.

*

  • @param build build

  • @param testId current test id

  • @param bean failed test info bean

  • @return string to be returned as the test's stacktrace.

*/
String formatTestForWeb(@NotNull final SBuild build, final int testId, @NotNull final FailedTestOutputBean bean);

/**

  • Returns true if the specified test output can be formatted by this extension and false otherwise

  • @param build build

  • @param bean failed test info bean

  • @return see above

*/
boolean canFormat(@NotNull final SBuild build, @NotNull final FailedTestOutputBean bean);
}


--
Pavel Sher

Edited by: Pavel Sher (Jetbrains) on Nov 20, 2008 7:07 PM

Edited by: Pavel Sher (Jetbrains) on Nov 20, 2008 7:08 PM

0
Comment actions Permalink

1. Never mind for that question. The following point I believe is a better way to handle this problem: determine if the formatter should format that message.

2. This is great change. I was thinking along the same line. It makes it better and more plugin developer friendly.
However this may not be enough. What if there isn't a unique formatter that could format a message? A TC server may have many different builds with different test frameworks for example. If a formatter plugin did not make a good job at properly uniquely identify its associated output and is too greedy, there may be conflicts rather difficult to trace.
What about making this explicit by adding an attribute to specify the type of message in the output tag? This way there isn't any chance for conflicts and since you are already forcing people to output a TC specific tags in the build log it should not be really hard to add that.

The formatter interface would have a method to return the type of output it formats. On registration TC make sure all formatters types are unique registered. When rendering the output of a build if the type is specified TC would look up the proper formatter based on it type and invokes that one.

3. How do I register my new formatter?
Given your upcoming design, how to I register my new formatter? What spring context files did you mention?
Also if I still have to call programmatically {{ExtensionHolder.registerExtension}} what values should I use for its arguments. {{sourceId}} for example, I suppose are ids of fixed extension points. Where do I find all these ids? What about the Extension instance? Am I supposed to create one an pass it in?
Without knowing if it is already there, can I package all this into a spring context file and have it included in the TC context startup?

4. Is the string returned by the {{formatTestForWeb}} supposed to be HTML?

5. How could I register a formatter style sheet/javascript to the header so it isn't replicated for every stack trace.

Jacques

0
Comment actions Permalink

2. This is great change. I was thinking along the same line. It makes it better and more plugin developer friendly.
However this may not be enough. What if there isn't a unique formatter that could format a message? A TC server may have many different builds with different test frameworks for example. If a formatter plugin did not make a good job at properly uniquely identify its associated output and is too greedy, there may be conflicts rather difficult to trace.


Right now server will log messages to the log that more than one formatter was able to format stacktrace and the first formatter will be choosen. I agree that there can be poorly written plugin but right now we will not be able to introduce types of stacktraces, since it is not an easy task.

3. How do I register my new formatter?


In the new EAP build you can define spring bean which will implement this interface and that's all that you need to do. You can also use ExtensionHolder and you can specify some unique text as source id, like your class name.

4. Is the string returned by the {{formatTestForWeb}} supposed to be HTML?


It can have HTML tags, they will be written as is.

5. How could I register a formatter style sheet/javascript to the header so it isn't replicated for every stack trace.


You can add page extension for PlaceId.ALL_PAGES_HEADER. Take a look here: http://www.jetbrains.net/confluence/display/TCD4/WebUIExtensions
Unfortunately documentation lacks samples so feel free to ask your questions here.

--
Pavel Sher

0
Comment actions Permalink

Thanks so much Pavel.
I will await the new EAP with great expectation :)

0
Comment actions Permalink

What can I say. You guys rock!
This incredible agility reminds me of the early days of IDEA. Unfortunately IDEA is too big now but boy, is it fun to have an intelligent and productive conversation with a developer in the morning and have a release with the agreed upon updates the very evening!
You can bet I am singing your praise to anyone that wants to hear me! :)

0
Comment actions Permalink

That is just a coincidence :) Actually we've tried to make EAP build from the beginning of the week.

--
Pavel Sher

0
Comment actions Permalink

I know that this was an extreme example but it really feel good.
From my experience, you guys have consistently been very responsive and eager to incorporate feedback. That is why I am impressed and signing your praise, not because you happened to deliver what I need on the very same day in what happens to be a major release first RC! I don't hold my breath to get that again any time soon. :)

0

Please sign in to leave a comment.