Endless Loop in Teamcity 9.0.5

Avatar
Markus Schneiders
Created

Hi,

our Teamcity Server 9.0.5. Enterprise Queues Build Jobs in an endless Loop after Failure.

Situation is as follows.

1) An Build starts and fails
2) On the Dashboard you can see Pending changes, but TC Restarts Build 1 without collecting the new Changes.

To Stop this, you need to Pause the Build and start a manual one. After that, everything works well

We use SVN 1.8.x

Could you please investigate in this, cause this is very annnoying

Kind Regards
Markus

Votes

0

Share

6 comments
Avatar
Alina Mishina
Created

Hi Markus,

Please attach screenshot showing the problem, full build log and teamcity-server.log file.

0
Avatar
Markus Schneiders
Created

Hi,

Screenshot :
TeamcityLoop.png

Teamcity Server Log
[2015-07-01 11:52:12,260]   INFO - ity.BuildQueuePriorityOrdering - New item Queued build {Build promotion {promotion id=657974,configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}}, item id=657974} with weight 0.00 inserted at the default position 7 in the end of the queue
[2015-07-01 11:52:12,267]   INFO - tbrains.buildServer.ACTIVITIES - Build added to queue; Queued build {Build promotion {promotion id=657974, configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}, modificationId=53168, revisions=["Trunk - BuildSystem" {instance id=15000, parent internal id=11, parent id=BuildSystem, description: "svn: svn://svngb10/addone/trunk/tools/BuildSystem"}: 94256_2015/06/29 11:39:23 +0200,"Incredibuilder" {instance id=15056, parent internal id=255, parent id=Tools_Incredibuilder4x_Incredibuilder, description: "svn: svn://svngb10/tools/Tools/Incredibuilder/Branch/4.x"}: 9935,"paexec" {instance id=15057, parent internal id=13, parent id=paexec, description: "svn: svn://svngb10/addone/trunk/tools/paexec"}: 87188,"signtool" {instance id=15058, parent internal id=12, parent id=signtool, description: "svn: svn://svngb10/addone/trunk/tools/signtool"}: 87188,"Trunk - BOMFC" {instance id=15064, parent internal id=17, parent id=BOMFC, description: "svn: svn://svngb10/addone/trunk/BOMFC"}: 94351,"Trunk - lib-3rdParty" {instance id=15114, parent internal id=7, parent id=TrunkLib3rdParty, description: "svn: svn://svngb10/addone/trunk/lib-3rdParty"}: 94332_2015/06/30 17:25:19 +0200]}, triggered by "Subversion"}
[2015-07-01 11:52:12,432]   INFO - tbrains.buildServer.ACTIVITIES - Build started; BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056} #5196 {build id=657974, promotion id=657974, history=false, agent="BuildAgent-VM6-01" {id=30, host=172.16.12.41:9094, agentTypeId=30, pool=Trunk Build}, triggered by "Subversion" (##vcsName='svn'). Started Wed Jul 01 11:52:12 CEST 2015}
[2015-07-01 11:52:12,434]   INFO -   jetbrains.buildServer.SERVER - Build removed from queue (started): 657974
[2015-07-01 11:52:12,436]   INFO -   jetbrains.buildServer.SERVER - Flush queue finished, number of builds started: 1
[2015-07-01 11:52:12,770]   INFO -   jetbrains.buildServer.SERVER - Running build saved to DB: build id=657974, promotion id=657974
[2015-07-01 11:52:35,238]   INFO -   jetbrains.buildServer.SERVER - Dependency "657977 => 657976" replaced with "657977 => 657269"
[2015-07-01 11:52:35,281]   INFO - ity.BuildQueuePriorityOrdering - New item Queued build {Build promotion {promotion id=657977,configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}}, item id=657977} with weight 0.00 inserted at the default position 7 in the end of the queue
[2015-07-01 11:52:35,285]   INFO - tbrains.buildServer.ACTIVITIES - Build added to queue; Queued build {Build promotion {promotion id=657977, configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}, modificationId=53168, revisions=["Trunk - BuildSystem" {instance id=15000, parent internal id=11, parent id=BuildSystem, description: "svn: svn://svngb10/addone/trunk/tools/BuildSystem"}: 94256_2015/06/29 11:39:23 +0200,"Incredibuilder" {instance id=15056, parent internal id=255, parent id=Tools_Incredibuilder4x_Incredibuilder, description: "svn: svn://svngb10/tools/Tools/Incredibuilder/Branch/4.x"}: 9935,"paexec" {instance id=15057, parent internal id=13, parent id=paexec, description: "svn: svn://svngb10/addone/trunk/tools/paexec"}: 87188,"signtool" {instance id=15058, parent internal id=12, parent id=signtool, description: "svn: svn://svngb10/addone/trunk/tools/signtool"}: 87188,"Trunk - BOMFC" {instance id=15064, parent internal id=17, parent id=BOMFC, description: "svn: svn://svngb10/addone/trunk/BOMFC"}: 94351,"Trunk - lib-3rdParty" {instance id=15114, parent internal id=7, parent id=TrunkLib3rdParty, description: "svn: svn://svngb10/addone/trunk/lib-3rdParty"}: 94332_2015/06/30 17:25:19 +0200]}, triggered by "Subversion"}
[2015-07-01 11:52:35,341]   INFO - tbrains.buildServer.ACTIVITIES - Build started; BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056} #5197 {build id=657977, promotion id=657977, history=false, agent="BuildAgent-VM7-01" {id=31, host=172.16.12.42:9094, agentTypeId=31, pool=Trunk Build}, triggered by "Subversion" (##vcsName='svn'). Started Wed Jul 01 11:52:35 CEST 2015}
[2015-07-01 11:52:35,343]   INFO -   jetbrains.buildServer.SERVER - Build removed from queue (started): 657977
[2015-07-01 11:52:35,344]   INFO -   jetbrains.buildServer.SERVER - Flush queue finished, number of builds started: 1
[2015-07-01 11:52:35,595]   INFO -   jetbrains.buildServer.SERVER - Running build saved to DB: build id=657977, promotion id=657977
[2015-07-01 11:53:21,690]   INFO -   jetbrains.buildServer.SERVER - Dependency "657980 => 657979" replaced with "657980 => 657269"
[2015-07-01 11:53:21,710]   INFO - ity.BuildQueuePriorityOrdering - New item Queued build {Build promotion {promotion id=657980,configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}}, item id=657980} with weight 0.00 inserted at the default position 7 in the end of the queue
[2015-07-01 11:53:21,714]   INFO - tbrains.buildServer.ACTIVITIES - Build added to queue; Queued build {Build promotion {promotion id=657980, configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}, modificationId=53168, revisions=["Trunk - BuildSystem" {instance id=15000, parent internal id=11, parent id=BuildSystem, description: "svn: svn://svngb10/addone/trunk/tools/BuildSystem"}: 94256_2015/06/29 11:39:23 +0200,"Incredibuilder" {instance id=15056, parent internal id=255, parent id=Tools_Incredibuilder4x_Incredibuilder, description: "svn: svn://svngb10/tools/Tools/Incredibuilder/Branch/4.x"}: 9935,"paexec" {instance id=15057, parent internal id=13, parent id=paexec, description: "svn: svn://svngb10/addone/trunk/tools/paexec"}: 87188,"signtool" {instance id=15058, parent internal id=12, parent id=signtool, description: "svn: svn://svngb10/addone/trunk/tools/signtool"}: 87188,"Trunk - BOMFC" {instance id=15064, parent internal id=17, parent id=BOMFC, description: "svn: svn://svngb10/addone/trunk/BOMFC"}: 94351,"Trunk - lib-3rdParty" {instance id=15114, parent internal id=7, parent id=TrunkLib3rdParty, description: "svn: svn://svngb10/addone/trunk/lib-3rdParty"}: 94332_2015/06/30 17:25:19 +0200]}, triggered by "Subversion"}
[2015-07-01 11:53:21,815]   INFO - tbrains.buildServer.ACTIVITIES - Build started; BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056} #5198 {build id=657980, promotion id=657980, history=false, agent="BuildAgent-VM5-02" {id=28, host=172.16.10.212:9095, agentTypeId=28, pool=Trunk Build}, triggered by "Subversion" (##vcsName='svn'). Started Wed Jul 01 11:53:21 CEST 2015}
[2015-07-01 11:53:21,818]   INFO -   jetbrains.buildServer.SERVER - Build removed from queue (started): 657980
[2015-07-01 11:53:21,820]   INFO -   jetbrains.buildServer.SERVER - Flush queue finished, number of builds started: 1
[2015-07-01 11:53:22,502]   INFO -   jetbrains.buildServer.SERVER - Running build saved to DB: build id=657980, promotion id=657980
[2015-07-01 11:53:44,297]   INFO -   jetbrains.buildServer.SERVER - Dependency "657983 => 657982" replaced with "657983 => 657269"
[2015-07-01 11:53:44,316]   INFO - ity.BuildQueuePriorityOrdering - New item Queued build {Build promotion {promotion id=657983,configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}}, item id=657983} with weight 0.00 inserted at the default position 7 in the end of the queue
[2015-07-01 11:53:44,320]   INFO - tbrains.buildServer.ACTIVITIES - Build added to queue; Queued build {Build promotion {promotion id=657983, configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}, modificationId=53168, revisions=["Trunk - BuildSystem" {instance id=15000, parent internal id=11, parent id=BuildSystem, description: "svn: svn://svngb10/addone/trunk/tools/BuildSystem"}: 94256_2015/06/29 11:39:23 +0200,"Incredibuilder" {instance id=15056, parent internal id=255, parent id=Tools_Incredibuilder4x_Incredibuilder, description: "svn: svn://svngb10/tools/Tools/Incredibuilder/Branch/4.x"}: 9935,"paexec" {instance id=15057, parent internal id=13, parent id=paexec, description: "svn: svn://svngb10/addone/trunk/tools/paexec"}: 87188,"signtool" {instance id=15058, parent internal id=12, parent id=signtool, description: "svn: svn://svngb10/addone/trunk/tools/signtool"}: 87188,"Trunk - BOMFC" {instance id=15064, parent internal id=17, parent id=BOMFC, description: "svn: svn://svngb10/addone/trunk/BOMFC"}: 94351,"Trunk - lib-3rdParty" {instance id=15114, parent internal id=7, parent id=TrunkLib3rdParty, description: "svn: svn://svngb10/addone/trunk/lib-3rdParty"}: 94332_2015/06/30 17:25:19 +0200]}, triggered by "Subversion"}
[2015-07-01 11:53:44,421]   INFO - tbrains.buildServer.ACTIVITIES - Build started; BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056} #5199 {build id=657983, promotion id=657983, history=false, agent="BuildAgent-VM8-01" {id=33, host=172.16.12.43:9094, agentTypeId=33, pool=Trunk Build}, triggered by "Subversion" (##vcsName='svn'). Started Wed Jul 01 11:53:44 CEST 2015}
[2015-07-01 11:53:44,446]   INFO -   jetbrains.buildServer.SERVER - Build removed from queue (started): 657983
[2015-07-01 11:53:44,448]   INFO -   jetbrains.buildServer.SERVER - Flush queue finished, number of builds started: 1
[2015-07-01 11:53:44,849]   INFO -   jetbrains.buildServer.SERVER - Running build saved to DB: build id=657983, promotion id=657983
[2015-07-01 11:54:08,329]   INFO -   jetbrains.buildServer.SERVER - Dependency "657986 => 657985" replaced with "657986 => 657269"
[2015-07-01 11:54:08,348]   INFO - ity.BuildQueuePriorityOrdering - New item Queued build {Build promotion {promotion id=657986,configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}}, item id=657986} with weight 0.00 inserted at the default position 7 in the end of the queue
[2015-07-01 11:54:08,352]   INFO - tbrains.buildServer.ACTIVITIES - Build added to queue; Queued build {Build promotion {promotion id=657986, configuration=BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056}, modificationId=53168, revisions=["Trunk - BuildSystem" {instance id=15000, parent internal id=11, parent id=BuildSystem, description: "svn: svn://svngb10/addone/trunk/tools/BuildSystem"}: 94256_2015/06/29 11:39:23 +0200,"Incredibuilder" {instance id=15056, parent internal id=255, parent id=Tools_Incredibuilder4x_Incredibuilder, description: "svn: svn://svngb10/tools/Tools/Incredibuilder/Branch/4.x"}: 9935,"paexec" {instance id=15057, parent internal id=13, parent id=paexec, description: "svn: svn://svngb10/addone/trunk/tools/paexec"}: 87188,"signtool" {instance id=15058, parent internal id=12, parent id=signtool, description: "svn: svn://svngb10/addone/trunk/tools/signtool"}: 87188,"Trunk - BOMFC" {instance id=15064, parent internal id=17, parent id=BOMFC, description: "svn: svn://svngb10/addone/trunk/BOMFC"}: 94351,"Trunk - lib-3rdParty" {instance id=15114, parent internal id=7, parent id=TrunkLib3rdParty, description: "svn: svn://svngb10/addone/trunk/lib-3rdParty"}: 94332_2015/06/30 17:25:19 +0200]}, triggered by "Subversion"}
[2015-07-01 11:54:08,459]   INFO - tbrains.buildServer.ACTIVITIES - Build started; BO trunk :: Build BOMFC Debug x64/win32 IBuild {id=BoTrunk_BuildBomfcDebugX64win32IBuild, internal id=bt1056} #5200 {build id=657986, promotion id=657986, history=false, agent="BuildAgent-VM2-01" {id=21, host=172.16.10.178:9094, agentTypeId=21, pool=Trunk Build}, triggered by "Subversion" (##vcsName='svn'). Started Wed Jul 01 11:54:08 CEST 2015}
[2015-07-01 11:54:08,460]   INFO -   jetbrains.buildServer.SERVER - Build removed from queue (started): 657986
[2015-07-01 11:54:08,463]   INFO -   jetbrains.buildServer.SERVER - Flush queue finished, number of builds started: 1
[2015-07-01 11:54:08,711]   INFO -   jetbrains.buildServer.SERVER - Running build saved to DB: build id=657986, promotion id=657986

BuildLog From Agent.

It is very strange, that there is no Build-Log on the Agent for this Period of Time.

[2015-07-01 11:48:37,296] out - ----------------------------------------- [ BO trunk::Build BOMFC Release x64/win32 IBuild #4544 (buildId = 657817) ] -----------------------------------------
[2015-07-01 11:48:37,297] out - Starting: C:\Windows\system32\cmd.exe /c IncrediBuilder.exe
[2015-07-01 11:48:37,297] out - in directory: D:\BuildAgent2\work\590028eb5d469b4\IncrediBuilder
[2015-07-01 11:48:37,299] err - C:\Windows\system32\cmd.exe /c IncrediBuilder.exe
[2015-07-01 11:48:38,380] out - ##teamcity[message text='Konfiguration : Release win32' status='NORMAL' timestamp='2015-07-01T09:48:38.358+0000' flowId='1']
[2015-07-01 11:48:38,380] out - ##teamcity[message text='Solution : D:\BuildAgent2\work\590028eb5d469b4\IncrediBuilder\..\src\bm\bmmfc.sln' status='NORMAL' timestamp='2015-07-01T09:48:38.378+0000' flowId='1']
[2015-07-01 11:48:38,380] out - ##teamcity[message text='Workindir : D:\BuildAgent2\work\590028eb5d469b4\IncrediBuilder\..\src\bm' status='NORMAL' timestamp='2015-07-01T09:48:38.378+0000' flowId='1']
[2015-07-01 11:48:38,380] out - ##teamcity[message text='Mail Empf„nger : bo@inform-software.com;addone-qs@inform-software.com' status='NORMAL' timestamp='2015-07-01T09:48:38.378+0000' flowId='1']
[2015-07-01 11:48:38,381] out - ##teamcity[message text='Mail Server : gs1-ex-3' status='NORMAL' timestamp='2015-07-01T09:48:38.379+0000' flowId='1']
[2015-07-01 11:48:38,405] out - ##teamcity[flowStarted parent='32768' timestamp='2015-07-01T09:48:38.404+0000' flowId='-1073741824']
[2015-07-01 11:48:38,405] out - ##teamcity[message text='Aufruf Parameter : "D:\BuildAgent2\work\590028eb5d469b4\IncrediBuilder\..\src\bm\bmmfc.sln" /CFG="Release||win32" /BEEP' status='NORMAL' timestamp='2015-07-01T09:48:38.404+0000' flowId='-1073741824']
[2015-07-01 11:48:38,456] out - ##teamcity[message text='' status='NORMAL' timestamp='2015-07-01T09:48:38.455+0000' flowId='-1073741824']
[2015-07-01 11:48:38,456] out - ##teamcity[message text='-----------------------------------------------------------------' status='NORMAL' timestamp='2015-07-01T09:48:38.455+0000' flowId='-1073741824']
[2015-07-01 11:48:38,457] out - ##teamcity[message text='Build Acceleration Console 6.0 (build 1580)' status='NORMAL' timestamp='2015-07-01T09:48:38.455+0000' flowId='-1073741824']
[2015-07-01 11:48:38,457] out - ##teamcity[message text='Copyright (C) 2001-2015 Xoreax Software Ltd. All rights reserved.' status='NORMAL' timestamp='2015-07-01T09:48:38.455+0000' flowId='-1073741824']
[2015-07-01 11:48:38,457] out - ##teamcity[message text='-----------------------------------------------------------------' status='NORMAL' timestamp='2015-07-01T09:48:38.456+0000' flowId='-1073741824']
[2015-07-01 11:48:38,458] out - ##teamcity[message text='' status='NORMAL' timestamp='2015-07-01T09:48:38.456+0000' flowId='-1073741824']
[2015-07-01 11:48:38,458] out - ##teamcity[message text='' status='NORMAL' timestamp='2015-07-01T09:48:38.456+0000' flowId='-1073741824']
[2015-07-01 11:48:39,092] out - ##teamcity[message text='' status='NORMAL' timestamp='2015-07-01T09:48:39.091+0000' flowId='-1073741824']
[2015-07-01 11:48:39,092] out - ##teamcity[message text='Waiting for current build to complete...' status='NORMAL' timestamp='2015-07-01T09:48:39.091+0000' flowId='-1073741824']
[2015-07-01 11:49:57,363] out - Process exited with code 1
[2015-07-01 12:11:20,039] out - ----------------------------------------- [ BO trunk::C++ Unittests für BOMFC #2543 (buildId = 658031) ] -----------------------------------------
[2015-07-01 12:11:20,070] out - Starting: D:\BuildAgent2\plugins\dotnetPlugin\bin\JetBrains.BuildServer.MsBuildBootstrap.exe /workdir:D:\BuildAgent2\work\fc2c545504c9a669 /msbuildPath:C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe
[2015-07-01 12:11:20,070] out - in directory: D:\BuildAgent2\work\fc2c545504c9a669
[2015-07-01 12:11:20,164] err - D:\BuildAgent2\plugins\dotnetPlugin\bin\JetBrains.BuildServer.MsBuildBootstrap.exe /workdir:D:\BuildAgent2\work\fc2c545504c9a669 /msbuildPath:C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe
[2015-07-01 12:12:42,305] out - Process exited with code 0
0
Avatar
Alina Mishina
Created

Hi Markus,

Thank you for provided details. Unfortunately the reason is not clear from the logs, please create an issue in our tracker.

0
Avatar
Alina Mishina
Created

Hi Markus,

Could you please attach full build log and screenshot of the overview page (build results) of one of the Canceled builds?

0
Avatar
Markus Schneiders
Created

Hi Alina,

here the Screenshot
TeamcityLoopOverview.png

and here the Log

[11:54:30]Skip checking for changes - changes are already collected
[11:54:33]Clearing temporary directory: D:\BuildAgent2\temp\buildTmp
[11:54:33]Publishing internal artifacts
[11:54:33][Publishing internal artifacts] Sending using ArtifactsCachePublisher
[11:54:33][Publishing internal artifacts] Sending using WebPublisher
[11:54:33]Checkout directory: D:\BuildAgent2\work\590028eb5d469b4
[11:54:33]Updating sources: server side checkout
[11:54:33][Updating sources] Using vcs information from agent file: 590028eb5d469b4.xml
[11:54:33][Updating sources] Building incremental patch for VCS root: XXX; checkout rules: =>; revision: 9935 --> 9935
[11:54:33][Updating sources] Building incremental patch for VCS root: Trunk - lib-3rdParty; checkout rules: =>lib-3rdParty; revision: 94332_2015/06/30 17:25:19 +0200 --> 94332_2015/06/30 17:25:19 +0200
[11:54:33][Updating sources] Building incremental patch for VCS root: Trunk - yyyy; checkout rules: =>src/bm; revision: 94351 --> 94351
[11:54:33][Updating sources] Building incremental patch for VCS root: Trunk - BuildSystem; checkout rules: =>tools/BuildSystem; revision: 94256_2015/06/29 11:39:23 +0200 --> 94256_2015/06/29 11:39:23 +0200
[11:54:33][Updating sources] Building incremental patch for VCS root: paexec; checkout rules: =>tools/paexec; revision: 87188 --> 87188
[11:54:33][Updating sources] Building incremental patch for VCS root: signtool; checkout rules: =>tools/signtool; revision: 87188 --> 87188
[11:54:33][Updating sources] Repository sources transferred
[11:54:33]Resolving artifact dependencies (2m:22s)
[11:54:33][Resolving artifact dependencies] Started downloading files from <AO trunk :: Build AO Debug x64/win32 IBuild, build #2123 [id 657031]>
[11:55:57]
[Resolving artifact dependencies] 1214 files retrieved from <AO trunk :: Build AO Debug x64/win32 IBuild, build #2123 [id 657031]> for pattern [+:AODebug.zip!/bin=>bin
+:AODebug.zip!/include/AO=>include/AO
+:AODebug.zip!/pdb=>pdb]
[11:55:57][Resolving artifact dependencies] Started downloading files from <XX trunk :: Build XXCommon Debug x64/win32 IBuild, build #2653 [id 657269]>
[11:56:56][Resolving artifact dependencies] Artifact resolving process was interrupted.
[11:56:56]Build was interrupted. Artifacts will not be published for this build
[11:56:56]Build canceled

0
Avatar
Alina Mishina
Created

Sorry for delay. Could you also please attach full teamcity-server.log? If you do not want to share it publicly you can create an issue in the tracker and attach it visible for jetbrains-team only. Thank you in advance.

0

Please sign in to leave a comment.