TeamCity sometimes shuts down cloud agent machine and restarts build.

Hello,

We've got the following configuration:

  • Teamcity 2017.1.3 (build 46961)
  • Cloud profile in VMware vSphere
  • After the first build terminate condition

Sometimes the following is observed:

Build is going on machine then in one moment this machine is shutting down and deleting. Build is restarted on another cloud machine.

Previous build gets the following status:

Canceled

Canceled with comment: Agent removed
 
New build has the following trigger:
 
Triggered by: you; Re-added because of unexpected finish on 14 Aug 17 16:58

I've looked into teamcity-clouds.log and there is the following:


[2017-08-14 16:56:49,926]   INFO [uled executor 5] - l.instances.StopInstanceAction - Terminating instance: VmwareCloudInstance{myInstanceName='TeamCity Build Agent SSD + VSsp3. v7-4'myState='RUNNING'myStatusUpdateTime='Mon Aug 14 16:24:56 MSK 2017'}
, profile 'vSphere TeamCity Build Agents Pool + VS SP3'{id=vmw-3, projectId=_Root}, reason: After the first build
[2017-08-14 16:56:49,926]   INFO [uled executor 5] - r.impl.DBCloudStateManagerImpl - Image: VmwareCloudImage{myName='TeamCity Build Agent SSD + VSsp3. v7'}, profile: profile 'vSphere TeamCity Build Agents Pool + VS SP3'{id=vmw-3, projectId=_Root} was marked to CONTAIN agent
[2017-08-14 16:56:49,932]  DEBUG [uled executor 5] - .server.impl.CloudEventsLogger - CloudInstance terminate command called: profile 'vSphere TeamCity Build Agents Pool + VS SP3'{id=vmw-3, projectId=_Root}, VmwareCloudInstance{myInstanceName='TeamCity Build Agent SSD + VSsp3. v7-4'myState='RUNNING'myStatusUpdateTime='Mon Aug 14 16:24:56 MSK 2017'}
[2017-08-14 16:56:49,932]   INFO [uled executor 5] - ces.terminate.AgentRemoverImpl - Removing cloud agent Unregistered agent "TeamCity Build Agent SSD + VSsp3. v7-4" {id=1119} because: Instance has been terminated. VmwareCloudInstance{myInstanceName='TeamCity Build Agent SSD + VSsp3. v7-4'myState='RUNNING'myStatusUpdateTime='Mon Aug 14 16:24:56 MSK 2017'}
, profile 'vSphere TeamCity Build Agents Pool + VS SP3'{id=vmw-3, projectId=_Root}

Logs is stating that reason of termination is After the first build but this should not happen as it is first build on this cloud agent and this build was nowhere near its completion.

So is there any way to get rid of this behavior.

Thank you

Upd:

This behavior is observed several times a week and also was observed on previous versions 2017.1.2 and 10.0.2.

0
2 comments

It might be caused by the fact that you have some conditions on your CA. For example shutdown after first build. 

I noticed this warning with the same problem "Cloud image is not configured to stop agents gracefully." so check out if the agent is shut down correctly when guest os shutdown is initialized.

 

0

Hi, can you please create an issue at https://youtrack.jetbrains.com/issues/TW and attach teamcity-server.log and teamcity-clouds.log, so they both cover a period when issue is reproduced?

Logs can be attached privately.

Thank you.

0

Please sign in to leave a comment.