Multiple TC build agents on a single AWS EC2 instance connected with the Agent Cloud plugin generate new agent names for each run

Answered

We use the latest Team City available (10.0.4) installed on an EC2 instance and our build agents are located in the cloud too. Our Agent Plugin has two profiles in its configuration - one for a Win build and another for a Linux build agent. Each profile is configured to run only one instance (the limit is 1). We don't use AMIs, but rather existing instances, which contain the VCS checkouts, build files, etc.

One of these instances - the linux one - contains two identical running docker containers with actual TC build agents installed. And I think if the behavior described in the next paragraph isn't normal - that's probably because of this linux instance with two docker containers.

The problem with this setting is that each run of an instance creates a new agent name, something like "EC2-i-0e619d91-142", notice the last number, which is not part of the id of the AWS instance. So as a result we have circa 700 unauthorized build agents and this list isn't cleaned automatically anywhere. Moreover I can't find the place where TC stores this list and modify it manually. The only one option left is to either clean it using the web UI, which would be a horrible rut, or write some kind of a JS script which will automate this process..

Another issue is that sometimes "dead" agents stay in the Disconnected list which counts in the overall agents number - and the default limitation is 3 agents. And the cloud plugin doesn't even try / allow to start the cloud build agents, though they are always the same two instances. The only workaround for this situation we have found so far is to manually delete the stuck agents from the disconnected list.

So is this behavior expected? Could it be fixed? Can we at least modify some file and delete all the old generated names in bulk?

Is it possible to force TC to not create a new name for the agents each time? In the settings there is only the "prefix" option, which doesn't help at all.

Update: After more thorough testing it appears that the scheme with having several build agents on a single AWS plugin-launched instance isn't possible anymore - we used this scheme with success on TC8.1, but apparently the behavior of TC in this case has change since then.

What seems to happen now is this:

  1. a build is scheduled,
  2. TC(10) searches for available and suitable build agents,
  3. when it doesn't find them, TC checks the profiles in the Agent Cloud plugin, finds a suitable profile, launches the specified instance,
  4. two build agents connect instead of one expected, they are both authorized,
  5. the build starts on either one of them, this agent now is considered as non-idle,
  6. the other agent is still considered idle and after the termination idle time lapses, TC tries to kill the AWS instance,
  7. the idle agent disconnects, but the other agent which was building disconnects and dies too - because they are on the same "physical" machine!

It's a pity that we can't use this approach anymore with the new TeamCity version. Or is there some workaround maybe?

 

1 comment
Comment actions Permalink

Hi, answering your questions:

1) The problem with this setting is that each run of an instance creates a new agent name, something like "EC2-i-0e619d91-142".

TC cloud integration doesn't work with two agents instances running on the same virtual machine (for numerous reasons) and should actually authorize only the first one that comes. The number at the end means that this is a 142's instance created this way. Please use two EC2 instances instead.

2) "this list isn't cleaned automatically anywhere". 

https://youtrack.jetbrains.com/issue/TW-48773

0

Please sign in to leave a comment.