Teamcity fails to start at "Loading history of VCS changes..."

I'm having a problem with Teamcity currently. When our server restarts Teamcity fails to restart properly. In looking at the logs it seems to be hanging at the line "Loading history of VCS changes...".

[2015-05-27 12:15:14,659]   INFO -   jetbrains.buildServer.SERVER - Server configuration loaded
[2015-05-27 12:15:14,763]   INFO -   jetbrains.buildServer.SERVER - Start prefetching changes for 4 build configurations...
[2015-05-27 12:15:14,763]   INFO -   jetbrains.buildServer.SERVER - Loading history of VCS changes...
[2015-05-27 12:58:24,609]   INFO -  jetbrains.buildServer.STARTUP - Server shutdown has been requested

As you can see Teamcity is just hanging until I shut it down.

Some looking at other points in the log pointed me at the following error accessing MySql:

SQL error when doing: Executing  with parameters: [1, 50001] caused SQL error:
SQL query: select h.modification_id, h.user_name, h.description, h.change_date, h.vcs_root_id, h.version, h.display_version, h.changes_count, h.register_date from vcs_history h join vcs_root_instance v on (h.vcs_root_id = v.id) where h.modification_id >= ? and h.modification_id < ? order by modification_id
SQL exception: Communications link failure


If I try to run this on my SQL server (via HeidiSQL) it seems to fail if there are more than a certain number of rows returned (14561 to be precise) with error message "SQL Error (2027): Malformed packet".


As a temporary fix I have identified rwos in vcs_root_instance and flipped their ids to negative so that the join finds less than 14561 rows. However this doesn't seem to be a good long term solution (as demonstrated by the fact I've had to do this twice already).



I've looked around to see if I coudl find any settings on mysql that could be wrong but have found nothing that makes a difference. I've also found nothing to help me diagnose the error any further than this.


So I have two questions:

1) Are there settings on MySql that would cause this problem and if so how can I fix them? (I've already gone through the settings on http://confluence.jetbrains.com/display/TCD9/How+To...#HowTo...-ConfigureNewlyInstalledMySQLServer and ensured they are all set)
2) Is there a way through the software to clean up the vcs_history and vcs_root_instance tables properly? I've deleted unused VCS roots from the interface but it doesn't seem to get rid of them from the database.


Many thanks for any help on this. Its a pain in the neck having to go and modify this table to try to fix it whenever it comes up and sooner or later our VCS history will grow large enough that our current active VCS roots alone will cause this problem so it'd be nice to get it fixed.


Edit: I just realised I should probably mention that we are running version "9.0.3 (build 32334)"

2 comments
Comment actions Permalink

Hi Chris,

Sorry for delay in replying.
Please stop TeamCity server, then try killing all TeamCity specific queries in MySQL (queries started by TeamCity server user). If MySQL server is used by TeamCity only, you can also try restarting it. After that try starting server again. Does it help?
Do you have an antivirus software installed? If yes, then please try to uninstall it and check if it helps.
If nothing helps, then please create issue in out tracker and attach teamcity-server.log and several thread dumps collected while server in hanging stage.

0
Comment actions Permalink

I didn't think it needed to be said that I had tried clean starts several times. It seemed pretty obvious that the problem was some setting on MySQL since I could run some queries fine and others (such as the one that Teamcity was trying to run) would fail.

In the end I found that if I dumped the data out of one database and into another on a different machine that things seemed to work. So I decided to do a compare of the two config files and update values that were different. In doing so I updated the following:

  • tmp_table_size
    from
    5M
    to
    19M
  • myisam_sort_buffer_size
    from
    8M
    to
    30M
  • read_buffer_size
    from
    0
    to
    59K
  • read_rnd_buffer_size
    from
    0
    to
    256K
  • sort_buffer_size
    from
    0
    to
    256K
  • query_cache_type
    from
    1
    to
    0


Doing this fixed my issues. I'm sure not all of them are needed but I wasn't going to spend time reverting some until it broke again to find which were needed. If you can explain why these fixed my issue though then I would be very happy to hear.

0

Please sign in to leave a comment.