TeamCity task queue / Java issues
Hello,
In the last few days we have seen TeamCity hit a spinning state where it just shows 'Running' on the dashboard, and all builds are essentially blocked from running, even though there is an agent assigned. Those builds that have attempted to start (or finish) just sit there and do nothing.
I see (what I suspect to be Java issues) in the log.
WARN - jetbrains.buildServer.SERVER - Could not schedule finishing of the running build: #3354039 {build id=618673, buildTypeId='My_Build'
ERROR - SERVERexecutors.normalExecutor - Unable to put task to queue because maximum capacity exceeded: 3000; executor info: Thread factory "Normal
executor" {corePoolSize=14; maxPoolSize=50; activeCount=50}
We are running 64 bit Java with the memory bumped up: -Xmx10g -XX:ReservedCodeCacheSize=512m
We have:
41 active projects
283 build configurations
156 Build Agents
12 VCS roots
Steve
Please sign in to leave a comment.
Hi Steve,
could you please take a few thread dumps as described here: https://confluence.jetbrains.com/display/TCD10/Reporting+Issues#ReportingIssues-ServerThreadDump
Then send them to us using the submit a request button on top, along with the teamcity-vcs and teamcity-server.log files from the logs folder.
Which teamcity version are you using?
Hi Denis,
We are running 10.0.5 on our main instance (2017.2.3) on our dev instance. I have not rolled out the upgrade just yet, as if we archive a project, it still shows on the dashboard, which clutters is up for users (attached shows what I am talking about).
I am submitting the threaddumps and logs now
I read another issue similar to ours and it was suggested to add the following to internal.properties. Is this a good idea?
teamcity.threadpool.lpexecutor.threads.max=50
teamcity.threadpool.lpexecutor.queue.capacity=5000
Steve
Hi Denis,
The logs have been uploaded.
Hi Steve,
thanks, will take a look at the logs on monday (they're rather long and it's the end of the day here), but wanted to pass some feedback on first. Those parameters are usually suggested for large installations if this error happens. But if you take a closer look, you already have the threads at 50 max pool size, so either you already have a similar set of parameters set up, or this ones won't really help. There are several different executors with their own queues so we need to find the right one here. At this point, I'm afraid a server restart is going to be required (and it will be required anyway to get the new parameters to work), once set up. If after the restart, this values get reached fast again, please make sure to mention it for review.