Collect statistics about time spent waiting for available agents

Hi,

can you check whether this statistic works for you? https://confluence.jetbrains.com/display/TCD10/Statistic+Charts#StatisticCharts-TimeSpentinQueue

We want to do the same - to determine whether we have the right number of agents or if we need more.

The "TimeSpentInQueue" statistic doesn't cut it, unfortunately, because it tells you the *total* time a build spent in the queue for *any* reason. We want to isolate the time spent in the queue *waiting for an idle agent* i.e. it *could* run but isn't. It is a subtle but important distinction.

If you think about it, the reason a build is in the queue can change over time. Say you have a build that has some dependencies, and uses a Resource. The reason may go through these states:

- waiting for dependencies to be built
- waiting for resource XYZ
- waiting for an idle agent

The first two are normal behaviour (certainly the first is), but the latter one indicates that we have hit a constraint on #agents and maybe we need more. Hence we want to analyse the amount of time spent in that latter state i.e. "the build *could* run if you had somewhere to run it".

Under normal circumstances that value should be close to zero. If we could analyse that value over time, we can make a judgement call as to whether we invest in more agents or live with it if it only happens occasionally. Better still, we could monitor that value and raise alerts to the support team if we detect a backlog building up.

+1 for Jason's use case. Some data on how long a build spent waiting for an available agent would be very useful in helping the continuous delivery of our products by pinpointing if it's a bottleneck or not.

Hi Jason and Nicolas,

thanks for your input. I understand what you are looking for, but what is available right now can be found in the link I sent. I don't think it quite covers your use cases so I would suggest you to open a feature request in our tracker here: https://youtrack.jetbrains.com/issues/TW

In the meantime, an option is to have some watchdog that pulls for the current statistic, and also checks the reason for wait parameter. That way you can narrow down the time waiting by substracting the time it was waiting for other reasons, such as dependencies finishing, which is one of the main causes of waiting time.

In the meantime can we achieve this somehow using Rest API ?

As I understand, this is not natively supported yet (running 2019.1.2), and the way we're pulling that information is by counting the number of builds in the queue that have no dependencies.

We use:

/app/rest/builds?locator=state:queued,compatibleAgent:(pool:(id:{}))

to get the number of builds in the queue, then use the following to count the dependency for each build (using their ID):

/builds?locator=snapshotDependency:(to:(id:{})),defaultFilter:false&fields=count

Hope this helps.

Hi, was a ticket every created for this issue, would love to see how we can grab this data. 

We tend to use the api to pull queue info, then walk that to find the wait reason for each and report off those numbers. We run this query about once a minute.

However, when I watch a build in our queue it changes state regularly (for us it does this mainly because we have a process for tearing up and spinning down agents in the cloud so that we don't run into issues with old data/packages)

Anyway, was thinking another way would be to create a simple hourly build job that echos out the "time waiting in the queue" value. I'm not sure though if that data exists as a variable/param, maybe something like %teamcity.queuetime% or something. Does it?

Hi James,

you can find the request here: https://youtrack.jetbrains.com/issue/TW-56008

There is no parameter for time spent in the queue, as far as I am aware. If you have a use case for it besides working around not having the statistic directly available, please consider adding a comment on the issue mentioning it.

I'm adding this here because I found this question, not because I expect anyone to be waiting for the answer here!

https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206050589-REST-API-Ability-to-get-Time-Spent-in-the-Queue-

/app/rest/builds/id:17004136/statistics/

<properties count="36">
...
<property name="queueWaitReason:Build_dependencies_have_not_been_built_yet" value="224707"/>
<property name="queueWaitReason:Checking_for_changes_is_in_progress" value="4824"/>
<property name="queueWaitReason:Waiting_to_start_checking_for_changes" value="523"/>
...
<property name="TimeSpentInQueue" value="230093"/>
...
</properties>