Test History over time and over all Builds in Project

Hello,

I have a number of tests (around 250) in my project. The tests are split into several TeamCity builds that are grouped in a single TeamCity project.
Because around half of the tests are failing I want to identify the tests, that are constantly failing, those that are constantly OK and the ones, that are sometimes OK and sometimes failing and how they behave over time.
So my idea was a view/export of the history of all tests below the project in a 2D table of this kind:

Test run 1
run 2
run 3
... run N-2
run N-1
run N
test abc OK OK OK OK OK OK
test def OK OK FAIL FAIL OK OK
...
test uvw FAIL FAIL FAIL FAIL FAIL FAIL
test xyz FAIL OK OK n/a n/a n/a


Showing me all tests in one dimension and all runs in the other one. If could have this kind of export for excel it would be great, because then I would be able to do some statistical excel magic (find all tests having only FAILs in a row, or mostly FAILs in a row, only FAILs in the last 20 builds, etc....)


I could not find a way to export statistics over multiple builds (in time). I also could'n find a way to export statistics over multiple builds below a project. I did find a hint on how to view test history of a single test over multiple builds, but this won't help me.


So my question is, how can I achive such an export?
Does anyone has another idea how I can identify easily test that do fail often, and tests whose result changed over time (from OK -> FAILED).


Since I would need to generate this report continuously (like once or twice a month) it would be great to automate it.


Kind regards,
Thorsten

2 comments
Comment actions Permalink

Hi Thorsten,

There is no such build-in functionality in TeamCity. As for now you can create a script that will get information about tests using REST API and feel the table.
At the same time you can develop a custom plugin to provide the information. And you can even share the plugin with other TeamCity users.
We have the related feature request https://youtrack.jetbrains.com/issue/TW-17550, please watch and vote for it.

0
Comment actions Permalink

Hi Alina,

thanks for your suggestions.
Actually I've already done some first steps in creating a TeamCity plugin that generates that kind of report.
Currently it looks like this:
historyOfAllTests.jpg
Because I did this in my working time and I'm working on a customer project it's not that easy, to share this plugin/source. But I'll see what I can do.

Best regards,
Thorsten

0

Please sign in to leave a comment.