Build fails when collecting changes on branch with large number of changes

We are using Teamcity with Mercurial. Builds from our main branch, or smaller branches work fine, but one of our developers created a branch with several thousand changes which we cannot build. When we try, Teamcity gets stuck on ''Collecting changes in VCS roots' and after about 90 minutes, the build will fail with an error like this in the build log:

Failed to collect changes, error: Error collecting changes for VCS repository '"**************" {instance id=55, parent internal id=7, parent id=**********, description: "mercurial: http://*****************"}'
'hg --encoding UTF-8 log -v --style=C:\TeamCity\temp\teamcity4339914566734617118hg.log.template -r "ancestors(107b4f074817) - ancestors(a41f74b338e3)"' command failed.

During those 90 minutes, the server CPU hovers around 99% (Java VM processes taking up most of it) . I've tried recreating the VCS root and a clean checkout, but got the same result.

The agent doesn't seem to be working too hard, but when it finally fails we see entries like the following in the agent's wrapper.log:

INFO | jvm 1 | 2018/04/30 09:00:03 | [2018-04-30 09:00:03,474] WARN - ldServer.AGENT.PollingProtocol - Error while asking server for the next command by GET http://****************/app/agents/v1/commands/next. Will continue polling: java.net.SocketException: Connection reset (enable debug to see stacktrace)

INFO | jvm 1 | 2018/04/30 09:00:36 | [2018-04-30 09:00:36,129] INFO - buildServer.AGENT.registration - Registering on server via URL http://****************: AgentDetails{Name='***********', AgentId=5, BuildId=null, AgentOwnAddress='null', AlternativeAddresses=****************], Port=**********, Version='47175', PluginsVersion='47175-md5-cf0818f798cbc120c8a9133fdc117413', AvailableRunners=[Ant, cargo-deploy-runner, dotnet-tools-dupfinder, dotnet-tools-inspectcode, Duplicator, ftp-deploy-runner, FxCop, gradle-runner, Inspection, jb.nuget.installer, jb.nuget.pack, jb.nuget.publish, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, JPS, Maven2, MSBuild, NAnt, NUnit, rake-runner, SBT, simpleRunner, sln2003, smb-deploy-runner, ssh-deploy-runner, ssh-exec-runner, VisualStudioTest, VS.Solution], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='****************', PingCode='****************'}

INFO | jvm 1 | 2018/04/30 09:00:36 | [2018-04-30 09:00:36,239] WARN - buildServer.AGENT.registration - Failed to obtain server supported protocols via URL http://****************/app/agents/protocols: TeamCity Server is starting. Will try later.

INFO | jvm 1 | 2018/04/30 09:00:36 | [2018-04-30 09:00:36,239] WARN - buildServer.AGENT.registration - Error registering on the server via URL http://****************. Will continue repeating connection attempts. Details: server is in a maintenance mode

INFO | jvm 1 | 2018/04/30 09:01:38 | [2018-04-30 09:01:38,737] INFO - buildServer.AGENT.registration - Registering on server via URL http://****************: AgentDetails{Name='****************', AgentId=5, BuildId=null, AgentOwnAddress='null', AlternativeAddresses=****************], Port=****, Version='47175', PluginsVersion='47175-md5-cf0818f798cbc120c8a9133fdc117413', AvailableRunners=[Ant, cargo-deploy-runner, dotnet-tools-dupfinder, dotnet-tools-inspectcode, Duplicator, ftp-deploy-runner, FxCop, gradle-runner, Inspection, jb.nuget.installer, jb.nuget.pack, jb.nuget.publish, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, JPS, Maven2, MSBuild, NAnt, NUnit, rake-runner, SBT, simpleRunner, sln2003, smb-deploy-runner, ssh-deploy-runner, ssh-exec-runner, VisualStudioTest, VS.Solution], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='****************', PingCode='****************'}

I tried on different agents, but the problem really seems to be happening on the Teamcity server. It looks like eventually it just gets overwhelmed and the server dies, then automatically restarts.

Thinking it might just that the server wasn't powerful enough, I upgraded the VM it runs on - now it only gets to about 30% CPU max, but it fails the same way every time when attempting to build this branch. 

Is there any way I can get it to build? Skip the collecting of changes somehow, at least for the first build in this branch?

 

 

0
1 comment

Please check teamcity-vcs.log, does it explain why the 'hg log' command failed?

0

Please sign in to leave a comment.