Using TeamCity 9.0.1 I'm trying to find a reliable way to detect and automatically stop hanging builds.
I tried using a slightly modified version of the kill-billd plugin, which checks the return value of SRunningBuild::isProbablyHanging() periodically after exceeding a certain time. While the plugin works fine, and stops the build, the hanging build detection was ... slow.
I created a build configuration for testing purpose that simply executes 'sleep 30', and run it 5 times so that TeamCity assumes an average execution time of 30 seconds. Afterwards I changed it to run an endless loop (while true; sleep 1). While the build status showed properly, that the build was overtime, after running for a minute, it took a total of 8 hours! for TeamCity to mark it as hanging. Now the documentation isn't too specific about the underlying algorithm for detecting builds:
"A build is considered to be "hanging" if its run time significantly exceeds estimated average run time and the build has not send any messages since the estimation was exceeded."
The key word is significantly. I couldn't find any hint within the documentation if I could influence the meaning of significantly. Ideally I could set it to something like 15 or 30 minutes instead of 8 hours.