Custom test history query
Hello,
in order to optimize our test runner (non junit) I would like to extract a list of problematic tests for a given project (over all configuration).
I would like to do a query like
"all tests that ran at least X times" AND "ran at least for Y days" AND "didn't fail during the last Z days" AND "have a sucess rate greater than N%"
and passes this list as a property to the ANT runner.
I browsed through the TC API but I think I need some help to get started with this.
Please sign in to leave a comment.
TeamCity already forms such a list but on per build configuration basis. And this list can be accessed by the build too: http://www.jetbrains.net/confluence/display/TCD4/Risk+Tests+Reordering+in+Custom+Test+Runner
Isn't it suitable for you?
Thanks for this link, its certainly a good starting point. If I look at the tests listed in the recentlyFailedTests file it seems that those are the tests that failed in the previous build (same configuration).
What I would like to know are the problematic tests for the whole project (all configurations) and over a much longer time frame (~ 1 to 2 weeks).
Our complete testsuite runs for ~2 to 3 hours and we would like to have a configuration that only runs the problematic tests.
Without going into plugin-development I have now started to add the recentlyFailedTests-File as an artifact and will write a short script that collects those lists from all
configurations and merges them.
Hi Pavel,
I'm currently trying to write a plugin to collect the testlist.
What I actually want is to get the list like in the project's "Problematic Tests" TAB (with hours as config parameter).
I looked into the jsp files for this page and found a class named 'jetbrains.buildServer.serverSide.stat.TestFailureRate'
(which doesn't seem to be part of the Open-API, but inside server.jar).
Can this be used to get the list inside a plugin ? If yes, it would be great to have some code snippet that shows how to
get the testlist from a SRunningBuild instance.
Another question I have: When I got the list, how can I transfer this list to the agent as a local file (similar to the recently-failed file) ?
Thanks,
Stefan