Load on Gitorious server increased significantly since upgrade to 6.5.6

Since upgrading to the latest TeamCity build, load has increased hugely on our locally hosted Gitorious server and is causing it to perform very poorly. I have attached teamcity-vcs.log and also a graph showing how CPU on the Gitorious server has gone up hugely from pretty much exactly when the upgrade completed.

Please can you let me know what you think is causing this? This is quite an urgent issue.

Thanks,

Rob



Attachment(s):
gitorious_CPU_use.png
teamcity-vcs.log.zip
6 comments
Comment actions Permalink

...and I don't know if it's related, but we get a lot of these in our stdout_xxx.log. Not seen anyone else complain about it.

[2012-02-10 18:06:52,817]   WARN -   jetbrains.buildServer.SERVER - Agent 'AgentDetails{Name='buildagent-004', AgentId=null, BuildId=null, AgentAddress='null', AlternativeAddresses=[10.0.10.65], Port=9090, Version='18130', PluginsVersion='18130->SHA-1P1lbz5ePDKIIjLEwutIQOuoNq/g=&SHA-1eySbd+ajDWLOveGPjLDwIYCyXj8=', AvailableRunners=[Ant, dotnet-dupfinder, Duplicator, FxCop, gradle-runner, Inspection, Ipr, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, JPS, Maven2, MSBuild, MSTest, NAnt, NUnit, rake-runner, simpleRunner, sln2003, VS.Solution], AvailableVcs=[perforce, mercurial, jetbrains.git, git, svn, cvs], AuthorizationToken='0e644c1ecf9601589dbffd0c684be984'}' attempts to register unknown VCS checkout plugin: git

0
Comment actions Permalink

also, attached is a teamcity-vcs.log from one of the build agents



Attachment(s):
buildagent-teamcity-vcs.log.zip
0
Comment actions Permalink

Hi Rob,

since 6.5 there were no significant changes in number of requests to the remote git repositories.

Did you change any VCS-related settings after upgrade, like checking option "Clean all files before build" in the build configuration settings?

Error message with unknown VCS plugin 'git' migth be related to your problem. Did you install any 3rd-party teamcity-git integration plugin? If so - try to delete it and check if it helps.

Your agent vcs log shows a number of clean checkouts. You can reduce this number by using local mirrors option, as described in the documentation http://confluence.jetbrains.net/display/TCD65/Git+(JetBrains).

0
Comment actions Permalink

Hi Dmitry,

We upgraded from 5.1 to 6.5.6 (not from 6.5)
Can you advise on what changes have been made since 5.1 which may be causing this?

Also, we did not change any vcs related settings.

Thanks,

Rob

0
Comment actions Permalink

Basic model of interaction with git in 6.5.6 is the same as in 5.1. TeamCity server polls remote repositories with the checking for changes interval, which can be configured at VCS root settings or globally for the server. You use an agent-side checkout, in this case remote repositories are also contacted by build agents before every build.

I can see the following reasons of the increased load on gitorious server (excluding changes in vcs-related settings):

- number of vcs roots increased: in this case TeamCity server will contact gitorious more often

- duration of the build decreased (due to use of faster hardware for build agents or build script optimization): in this case there will be more builds per unit of time, therefore gitorious will be contacted more often by build agents

- number of agents increased: therefore more builds per unit of time and more clones done by build agents

Does any of them apply to your case?

If you have logs of gitorious server prior to TeamCity upgrade please check what is the source of increased load: is it TeamCity server or build agents? If it is server - is number of request increased for all VCS roots or just for some of them, and in latter case is anything special about such roots. If load comes from build agents - is it distributed evenly among agents?

If the load comes from TeamCity server, you can try to increase checking for changes interval. If it comes from build agents, consider using local mirrors option, or switch to server-side checkout (in this case more work will be done by TeamCity server, it will build patches for builds instead of delegating this work to git, but gitorious will be contacted only by the server).

0
Comment actions Permalink

Thanks Dimitry,

None of those apply. We didn't make any other changes apart from upgrade 5.1 -> 6.5.6

Unfortunately we don't have any logs from the previous version. I have implemented the local mirrors configuration and already reduced the vcs checking schedule. We shall monitor it and see how it goes.

0

Please sign in to leave a comment.