Floating benchmark for unit test time

Hi guys,

we learned that some of our code and relevant unit test grows slower over time. We would like to be warned about that.
We could use timeout attribute on each individual unit test and manualy manage the benchmark. But that's lot of labor and it's fragile.

So I consider option to build the benchmark as average of initial 20 builds on same TC agent.
Then I would like to fail the test if the floating average on last 10 builds is 10% worse than the benchmark.
If all of last 10 builds are 10% better than the benchmark, I update the benchmark.
(All the numbers would be configurable and are guesses for now)

I would like to learn if someone tried to build something like that in the past.

If not, how difficult it would be to build TC plugin and access the historic data in TC database.
Also store the bechmark per test.
If that's doable enough, I would create the plugin and publish it for comunity.
Othewise I'll just hack something inside the build script.

Please help me with work breakdown, so we can evaluate the effort.
Also links to relevant documentation would be appreciated.

Thanks a lot

Please sign in to leave a comment.