RFE: Limit number of simultaneously running builds vs. hanging agents

set limit to 1, agent disconnects in the evening while build is still running, build will have warning sign "probably hanging" but next scheduled build will NOT run before manually cancelling hanging build due to of max running builds = 1

proposal: do not count in hanging builds for max running builds

comments?

8 comments
Comment actions Permalink

Yann Cébron wrote:

set limit to 1, agent disconnects in the evening while build is still running, build will have warning sign "probably hanging" but next scheduled build will NOT run before manually cancelling hanging build due to of max running builds = 1
proposal: do not count in hanging builds for max running builds
comments?

The problem is that "hang detection" is not 100% reliable (because it is
done via something like "last 5 mins w/o any messages" heuristic), thus
it is completely possible that things can go wrong with come
configurations. Although in general this is doable.

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Comment actions Permalink

so would you consider it safe to skip counting builds hanging for longer than e.g. 1 hour continuously?

0
Comment actions Permalink

Yann Cébron wrote:

so would you consider it safe to skip counting builds hanging for longer than e.g. 1 hour continuously?


I suppose. Anyway, feel free to add feature request to our tracker.
We'll see how it fits.

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Comment actions Permalink

You can also disable hanging builds detection for the build configuration.

--
Pavel Sher

0
Comment actions Permalink

Yann,

Simple "not counting" for hanging builds does not seem a universal approach.

e.g. consider we have 4 agents and 2 projects, each has limit of simultaneous build set to 2.
Now a build gets a check-in that just hangs the build. With the suggested approach in a couple of build-times we will get all 4 agents running hanging builds and no other build can start.

Setting execution timeout for a build seems to be a recommended approach here.

0
Comment actions Permalink

Hello Yegor,

I agree and have execution timeout set on the build configuration, but it seems it is not applied to hanging builds.

Greetings,

Yann

0
Comment actions Permalink

Yann,

I agree and have execution timeout set on the build configuration, but it seems it is not applied to hanging builds.


I managed to reproduce the problem only when the agent disconnects abruptly. Please vote/watch: http://jetbrains.net/tracker/issue/TW-5771

Actually, there is one more issue that requests dropping the builds running on abruptly disconnected agents even if there is no execution timeout set. If you think this is the desired behavior, please vote.

0
Comment actions Permalink

Thanks, that is exactly my observation as well.

0

Please sign in to leave a comment.