Custom test history query

Hello,

in order to optimize our test runner (non junit) I would like to extract a list of problematic tests for a given project (over all configuration).

I would like to do a query like
   "all tests that ran at least X times" AND "ran at least  for Y days" AND "didn't fail during the last Z days" AND "have a sucess rate greater than N%"
and passes this list as a property to the ANT runner.

I browsed through the TC API but I think I need some help to get started with this.

3 comments
Comment actions Permalink

TeamCity already forms such a list but on per build configuration basis. And this list can be accessed by the build too: http://www.jetbrains.net/confluence/display/TCD4/Risk+Tests+Reordering+in+Custom+Test+Runner

Isn't it suitable for you?

0
Comment actions Permalink

Thanks for this link, its certainly a good starting point. If I look at the tests listed in the recentlyFailedTests file it seems that those are the tests that failed in the previous build (same configuration).

What I would like to know are the problematic tests for the whole project (all configurations) and over a much longer time frame (~ 1 to 2 weeks).
Our complete testsuite runs for ~2 to 3 hours and we would like to have a configuration that only runs the problematic tests.

Without going into plugin-development I have now started to add the recentlyFailedTests-File as an artifact and will write a short script that collects those lists from all
configurations and merges them.

0
Comment actions Permalink

Hi Pavel,

I'm currently trying to write a plugin to collect the testlist.

What I actually want is to get the list like in the project's "Problematic Tests" TAB (with hours as config parameter).

I looked into the jsp files for this page and found a class named 'jetbrains.buildServer.serverSide.stat.TestFailureRate'
(which doesn't seem to be part of the Open-API, but inside server.jar).

Can this be used to get the list inside a plugin ? If yes, it would be great to have some code snippet that shows how to
get the testlist from a SRunningBuild instance.

Another question I have: When I got the list, how can I transfer this list to the agent as a local file (similar to the recently-failed file) ?

Thanks,

Stefan

0

Please sign in to leave a comment.