Teamcity fails to start at "Loading history of VCS changes..."

Created May 27, 2015 14:13

I'm having a problem with Teamcity currently. When our server restarts Teamcity fails to restart properly. In looking at the logs it seems to be hanging at the line "Loading history of VCS changes...".

[2015-05-27 12:15:14,659]   INFO -   jetbrains.buildServer.SERVER - Server configuration loaded
[2015-05-27 12:15:14,763]   INFO -   jetbrains.buildServer.SERVER - Start prefetching changes for 4 build configurations...
[2015-05-27 12:15:14,763]   INFO -   jetbrains.buildServer.SERVER - Loading history of VCS changes...
[2015-05-27 12:58:24,609]   INFO -  jetbrains.buildServer.STARTUP - Server shutdown has been requested

As you can see Teamcity is just hanging until I shut it down.

Some looking at other points in the log pointed me at the following error accessing MySql:

SQL error when doing: Executing  with parameters: [1, 50001] caused SQL error:
SQL query: select h.modification_id, h.user_name, h.description, h.change_date, h.vcs_root_id, h.version, h.display_version, h.changes_count, h.register_date from vcs_history h join vcs_root_instance v on (h.vcs_root_id = v.id) where h.modification_id >= ? and h.modification_id < ? order by modification_id
SQL exception: Communications link failure

If I try to run this on my SQL server (via HeidiSQL) it seems to fail if there are more than a certain number of rows returned (14561 to be precise) with error message "SQL Error (2027): Malformed packet".

As a temporary fix I have identified rwos in vcs_root_instance and flipped their ids to negative so that the join finds less than 14561 rows. However this doesn't seem to be a good long term solution (as demonstrated by the fact I've had to do this twice already).

I've looked around to see if I coudl find any settings on mysql that could be wrong but have found nothing that makes a difference. I've also found nothing to help me diagnose the error any further than this.

So I have two questions:

1) Are there settings on MySql that would cause this problem and if so how can I fix them? (I've already gone through the settings on http://confluence.jetbrains.com/display/TCD9/How+To...#HowTo...-ConfigureNewlyInstalledMySQLServer and ensured they are all set)
2) Is there a way through the software to clean up the vcs_history and vcs_root_instance tables properly? I've deleted unused VCS roots from the interface but it doesn't seem to get rid of them from the database.

Many thanks for any help on this. Its a pain in the neck having to go and modify this table to try to fix it whenever it comes up and sooner or later our VCS history will grow large enough that our current active VCS roots alone will cause this problem so it'd be nice to get it fixed.

Edit: I just realised I should probably mention that we are running version "9.0.3 (build 32334)"

2 comments

Alina Mishina

Created June 05, 2015 12:10

Hi Chris,

Sorry for delay in replying.
Please stop TeamCity server, then try killing all TeamCity specific queries in MySQL (queries started by TeamCity server user). If MySQL server is used by TeamCity only, you can also try restarting it. After that try starting server again. Does it help?
Do you have an antivirus software installed? If yes, then please try to uninstall it and check if it helps.
If nothing helps, then please create issue in out tracker and attach teamcity-server.log and several thread dumps collected while server in hanging stage.

Chris Venus

Created June 05, 2015 15:04

I didn't think it needed to be said that I had tried clean starts several times. It seemed pretty obvious that the problem was some setting on MySQL since I could run some queries fine and others (such as the one that Teamcity was trying to run) would fail.

In the end I found that if I dumped the data out of one database and into another on a different machine that things seemed to work. So I decided to do a compare of the two config files and update values that were different. In doing so I updated the following:

```
tmp_table_size
```
from
```
5M
```
to
```
19M
```
```
myisam_sort_buffer_size
```
from
```
8M
```
to
```
30M
```
```
read_buffer_size
```
from
```
0
```
to
```
59K
```
```
read_rnd_buffer_size
```
from
```
0
```
to
```
256K
```
```
sort_buffer_size
```
from
```
0
```
to
```
256K
```
```
query_cache_type
```
from
```
1
```
to
```
0
```

Doing this fixed my issues. I'm sure not all of them are needed but I wasn't going to spend time reverting some until it broke again to find which were needed. If you can explain why these fixed my issue though then I would be very happy to hear.

Please sign in to leave a comment.