4.0.1 (build 8171) - Excessive CVS usage (very high load)

Hello,

Has anyone else experienced high load on cvs with version 4.0.1 of team city? I just upgraded from version 3 (3.1.2 build 6881) to 4.0.1 and i am unable to run any builds.

The overview page reports (this is one of many - 189 build configs --> 9 VCS roots -- all on CVS):

Failed for the root 'Business - HEAD' #67: No response from server.
« Hide details

jetbrains.buildServer.vcs.VcsException: org.netbeans.lib.cvsclient.connection.AuthenticationException: No response from server.
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsCommandExecutor.executeServerCommand(CvsCommandExecutor.java:106)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSettings.doFetchModules(CvsSettings.java:118)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSettings.getCvsModules(CvsSettings.java:106)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CheckoutRulesSettings.getModulePrefixes(CheckoutRulesSettings.java:42)
at jetbrains.buildServer.buildTriggers.vcs.cvs.RLogCommandExecutor.<init>(RLogCommandExecutor.java:30)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsChangesProvider.<init>(CvsChangesProvider.java:20)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSupport.collectRawChanges(CvsSupport.java:217)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSupport.extractCompletedChanges(CvsSupport.java:175)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSupport.collectBuildChanges(CvsSupport.java:132)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:37)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChanges(VcsChangesLoader.java:157)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:169)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:61)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:11)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.netbeans.lib.cvsclient.connection.AuthenticationException: No response from server.
at org.netbeans.lib.cvsclient.connection.PServerConnection.openConnection(PServerConnection.java:179)
at org.netbeans.lib.cvsclient.connection.PServerConnection.open(PServerConnection.java:97)
at org.netbeans.lib.cvsclient.RequestProcessor.openConnection(RequestProcessor.java:101)
at org.netbeans.lib.cvsclient.RequestProcessor.processRequests(RequestProcessor.java:88)
at org.netbeans.lib.cvsclient.command.checkout.ListModulesCommand.execute(ListModulesCommand.java:53)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsCommandExecutor.executeServerCommand(CvsCommandExecutor.java:96)
... 19 more
org.netbeans.lib.cvsclient.connection.AuthenticationException: No response from server.
at org.netbeans.lib.cvsclient.connection.PServerConnection.openConnection(PServerConnection.java:179)
at org.netbeans.lib.cvsclient.connection.PServerConnection.open(PServerConnection.java:97)
at org.netbeans.lib.cvsclient.RequestProcessor.openConnection(RequestProcessor.java:101)
at org.netbeans.lib.cvsclient.RequestProcessor.processRequests(RequestProcessor.java:88)
at org.netbeans.lib.cvsclient.command.checkout.ListModulesCommand.execute(ListModulesCommand.java:53)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsCommandExecutor.executeServerCommand(CvsCommandExecutor.java:96)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSettings.doFetchModules(CvsSettings.java:118)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSettings.getCvsModules(CvsSettings.java:106)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CheckoutRulesSettings.getModulePrefixes(CheckoutRulesSettings.java:42)
at jetbrains.buildServer.buildTriggers.vcs.cvs.RLogCommandExecutor.<init>(RLogCommandExecutor.java:30)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsChangesProvider.<init>(CvsChangesProvider.java:20)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSupport.collectRawChanges(CvsSupport.java:217)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSupport.extractCompletedChanges(CvsSupport.java:175)
at jetbrains.buildServer.buildTriggers.vcs.cvs.CvsSupport.collectBuildChanges(CvsSupport.java:132)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:37)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChanges(VcsChangesLoader.java:157)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:169)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:61)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:11)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595

Our poor CVS server is really being hammered. This also has the result of our other teamcity server reporting the same issue (a working version 3.0 teamcity install)...
Shutting down the version 4 server results in its companion server starting to function correctly again.

The server log has the same error as above.

If anyone could please help ... i'd appreciate it, else i'm looking at rolling back to version 3 (which i'm seriously not looking forward to).

Regards,
Kirk
14 comments
Comment actions Permalink

First of all please enable debug logging on your server: http://www.jetbrains.net/confluence/display/TCD4/Reporting+Issues

Also to reduce load on CVS you can do the following:
- use quiet period in VCS triggers
- increase checking for changes interval

Actually there were some optimization in the TeamCity CVS plugin. These optimization were done for the case when one VCS root is shared among different configurations with different checkout rules. In such setup TeamCity 4.0 should produce less load on CVS than 3.x version.

Could you please describe your VCS root or probably attach a screenshot with VCS root settings?

0
Comment actions Permalink

Hi Pavel,

Please find attached the screen shot of one of the roots. I've also taken of the server config settings.
Log file attached.

I hope this is a simple thing to solve.

I've set the default changes check interval to 300 and default cvs trigger quiet time to 30.

Regards,
Kirk



Attachment(s):
tc vc screenshots.rar
server logs 22 Jan 2009 205100.rar
0
Comment actions Permalink

Could you please check are there any error messages on the CVS server side? Probably there are some logs?

Default VCS trigger quiet period is a default setting only, it does not enable quiet period in VCS trigger. To enable quiet period you should go to the build configuration VCS trigger settings.

0
Comment actions Permalink

Also please start TeamCity server with this additional parameter:
-Dcvs.log.commands=true

Please refer to this document if you do not know how to specify this parameter to TeamCity server: http://www.jetbrains.net/confluence/display/TCD4/Server+Startup+Properties

0
Comment actions Permalink

I've applied a quiet time of 60 secs to all the cvs roots. I've updated the tomcat service and added the extra parameter. The
Attached are the updated logs.



Attachment(s):
server logs 23 Jan 2009 062400.rar
0
Comment actions Permalink

I'll see if i can get hold of some cvs server logs. Waiting for the cvs admin to get into work....

0
Comment actions Permalink

Pavel,

The cvs server has indicated no obvious errors. We shutddown our version 3 server to see if that solved the problem. It did not.

Performing a netstat on the cvs server (unix) indicates that our the version 4 teamcity correctly establishes a connection to the box. The status of these connections are 'CLOSE_WAIT'.

This leads us to believe there is something wrong with cvs on the teamcity version 4...
(As to why it affected the version 3 server i'm not sure....)

In the mean time, i'll need try and rollback to version 3....

0
Comment actions Permalink

To rollback to 3.x you need your backup. However you will lost data created after the upgrade to 4.0.

As I told in 4.0 there was a fix which might affect your configuration. As I see you are using checkout rules a lot. When CVS plugin is going to collect changes for one of your roots it combines all checkout rules from all of the build configurations. In your case combined checkout rule has 132 lines. In previous version TeamCity would issue 132 requests to CVS server (one request for each line specified in the checkout rules). In 4.0 the request is only one but it includes all directories. Probably this is too much for your server.

It would be better if you can wait for some time (several hours) and I will prepare a patch for you. I plan to split large requests to several smaller requests, the part size will be configurable via a system property. I think there are good chances that this fix will help you.

0
Comment actions Permalink

I've attached two jar files to this message. You should put them to the WEB-INF/plugins/cvs/server directory instead of original jars (please backup original jars to be able to revert in case of any failure). By default this plugin has limit of number of directories/files set to 80, i.e. each request to the CVS server will contain no more than 80 directories.

If this will not help try to lower the limit.For this you should restart server with the following additional parameter: -Dteamcity.cvs.maxRequest.size=<num>
Replace <num> with desirable number of directories.



Attachment(s):
cvs-support.jar
cvs-common.jar
0
Comment actions Permalink

Hi Pavel,

I think this might be solved but i'll need to monitor it. Will update on monday if everything is still ok by then.
The problem was a combination of the companion version 3 teamcity server and the linux cvs server refusing any more connections from the teamcity 4 server.

The high cpu usage was from the tc version 3 server... large number of connections..

Connections were being denied to the version 4 server though... (init.d max connetions per ip on the linux box - was low, set to 10)

hence the exception...

jetbrains.buildServer.vcs.VcsException: org.netbeans.lib.cvsclient.connection.AuthenticationException: No response from server
.
.
.
.

So, a bit of a wild goose chase i'm afraid...

I've got the version 4 server now working... for the moment..

I will still be applying your fixes and will be reporting back on the cvs performance..

Thanks,
--Kirk

0
Comment actions Permalink

Well, I think you should consider ability to upgrade all your servers to 4.0.1

0
Comment actions Permalink

Oh definately!

The release manager insists we do the upgrade during a quiey period though.
Have a release next week and we need our v3 team city server for that... with its respective config.

Thanks for your help... and quick response.
BTW, excellent product! All the developers and managers love it...

0
Comment actions Permalink

Then I would suggest to wait till TeamCity 4.0.2 is released. We hope to upload it next week.

0

Please sign in to leave a comment.