Run Custom Build and other operations extreme slowness
Hi,
I'm having some extreme performance problems after migrating to a new server.
For instance opening up the Run Custom Build dialog on each build conf takes a few minutes to complete, mostly just for the first time though.
TeamCity has taken a few automatic thread dumps in the mean time, due to extemely long processing times. I'll try attaching those here.
Please sign in to leave a comment.
I couldn't find a way to attach the files so here's a Dropbox link: https://www.dropbox.com/sh/npm83quj2yg5ueh/AAAJU4KjM5q5vRLZtFe_wkKSa?dl=0
Hi,
from the logs you have shared, it seems that the server is constantly retrying SQL requests, but the logs don't really show why. While the request that originates the SQL queries has been going on for over a minute at the time of those dumps, the SQL requests have actually happened at that same second the dump was taken, so it's also hard to see this as an SQL issue. Another usual culprit is memory, and you are using 1Gb with a 64 bit JVM, which is pretty much what we ship teamcity with for test purposes, 32bit with 512Mb, but the dumps don't seem to imply that there is intensive usage of it either. I mention this two issues as you might want to keep them monitored just in case.
At this stage, with the main usual culprits seemingly out of the way, I can only recommend checking for possible error messages on the teamcity-server.log file or the browser console. If you need help with double checking those, please send the file to us via the "Submit a request" button on top of this page.
Sadly, the teamcity-server.log has nothing. Neither does the browser console.
I just had another extra long opening, and added new thread dumps under https://www.dropbox.com/sh/npm83quj2yg5ueh/AAAJU4KjM5q5vRLZtFe_wkKSa?dl=0
Look for those that end with longRequest-3.txt
Could you please look through those? Perhaps it really is some SQL issue?
I've tried profiling the SQL server, but there seem to have been no long running requests to my TeamCity database.
Now, interestingly enough, after switching TeamCity back to using filesystem as database, it started working with proper speed again...
Hi,
I've been going through the dumps and I can see how a UI request for a custom build takes north of a minute and every time it's doing some SQL related stuff. While the sql server might be answering in time, maybe there is an issue with the network? proxy servers, firewalls, issues with connectivity that might be delaying this. I/O operations would also influence the performance of the server. It doesn't seem to be strictly related to this, but might be worth checking out, as every other usual culprit is already out of the question.
Yeah, I think we already narrowed it down to the networking problems. It's a new server, and after I've set up our applications on it, we noticed the crap performance of the database connection aswell... So it's not anything related to TeamCity in particular.
Thanks for you help, though!