Flagging sudden performance changes

It would be really nice if TeamCity could notice when some change has significantly impacted the run time of any given test or set of tests over the established running averages.

4 comments
Comment actions Permalink

Hello Dave,

  Thanks for the suggestion!

  The possible problem with this request is that duration of the test depends not only on the test itself, but also on

  - specific build agent the test is running on
  - load of the build agent when test is running
  - availability of the external resources for the test (if there are any)

  What do you think?

  Kind regards,
  KIR

0
Comment actions Permalink

KIR,

You are, of course, correct on all counts.  Particularly with regard to load on an agent.  Detecting changes related to agent load probably isn't possible, especially if it is a VM competing for host resources.

But... if a test has been run many times you should have enough data to compensate for most, if not all, of those variables.  You have access to the average test time, standard deviation, and frequency/magnitude of outliers for each test on each agent.  So I would think it should be possible to flag possible non-characteristic behavior.  If a test has taken between <1 second for every one of the last 100 runs on that agent and suddenly it takes 6 - something is probably worth looking at.  Perhaps the first time it happens is a fluke.  But if the next 10 runs all take 6 seconds then clearly something has changed.  Yes, what changed COULD be something external (addition of multiple new VM's to the host, etc.), but the odds are pretty good that it isn't and some kind of alert is in order.

Thanks for the reply and the consideration.

Dave

0
Comment actions Permalink

Hello Dave,

  Thanks for sharing your thoughts.

  You're right that we have some statistical information about test runs over the time (or, we can calculate it).
  But, there are still things which could change internally, when statistics won't work.

  For instance, I can change a test so it will run longer. Or, I'll change thresholds for test duration.
  If I rely on statistic only, I'll have to change test name to collect new duration data, because old values will be non-actual.

  So, there should be a way to set/change anchor value for the test duration.

  For statistics approach, please see issue http://youtrack.jetbrains.net/issue/TW-16680
  You may also want to vote for http://youtrack.jetbrains.net/issue/TW-10513

  Thanks!
  KIR

0
Comment actions Permalink

Unlike a test failure, a change in performance could be a "warning". It doesn't need to be managed within the nice TC error pipeline. All we need is an alert of some kind, so that we can take action.


Anything would be better than nothing! Keep things simple for rev1 - a notification that we can subscribe to. Have it flag anything that looks unusual and leave it at that.

If the notification includes durations history then we can apply our own filters.


> For instance, I can change a test so it will run longer. Or, I'll change thresholds for test duration.

If I change a test so that it runs longer or shorter, a simple workaround is to rename the test.


-Erik
0

Please sign in to leave a comment.