Performance degradation after upgrading to 2017.1.4
After upgrading to 2017.1.4 we are seeing significant performance degradation in running our teamcity server.
Startup time of the server after a restart is 30-60 mins. we are having to reboot weekly as ui is hanging.
We have some builds that have a high number of tests ran in parallel (>68,000, come from large number of auto generated tests). Build times have gone from 2-3 min to >7mins. Also the number of tests report has greatly increased, previously teamcity under reported the number of tests but the new version 2017.1.4 correctly reports the large number of tests.
There does seem to be a large amount of activity on the test_names and test_info tables when looking at db io, they are also very large, test_names has 13697676 rows, and test_info 68664498 rows. i have already tried increasing memory team-city to 4gb and increasing mysql buffer_pool to 5gb.
Setup:
Teamcity 2017.1.4 running in docker
Mysql 5.7.17 also running in a docker container
Builds running in java 6,7,8 using gradle 2.13
Any help appreciated.
Please sign in to leave a comment.
Hi Andy, thanks for your report and sorry for the delay.
Please, check this article on how to debug and report issues with server performance. I am unaware of other reports of such drastic changes, so if you could provide some server thread dumps of the server starting up and some of the agents during the build, along with the corresponding build log, that would probably be useful. Please use the "Submit a request" button to send those files to us.