EC2 Integration Best Practices

Hi All,

We were looking into the possibility of having our build agents in the cloud and the master build server still reside as a physical machine in our office.

I was wondering what the best practices were for this, as I was a bit confused how communication between the two occur.

Some questions I had were:

  • How does communication between the ec2 build agents and the local master server work? If ec2 instances are private addresses only how does it communicate? Do you need to use elastic IP's for each of the ec2 build agents so an address can be NAT'd? Do you need to establish a vpn tunnel between the local site and the ec2 build agents?
  • Can agents be started up automatically when there is a build queue and shut down after the build is done?


I read over - http://www.jetbrains.com/teamcity/features/amazon_ec2.html a bit, but from the ec2 side things don't make much sense to me. Has anyone run into this situation, and found a working solution?

Thanks,
- Josh

14 comments

To make build agents from cloud work with TeamCity you will need to make TeamCity available to build agent by HTTP. TeamCity server uses HTTP to communicate to a build agent back. For this communication you need to have an HTTP opened on the build agent side. You may configure security group EC2 to allow only connections from your server's public IP to opened build agent port. I believe you do not need Elastic IP feature for build agents. On connection to server build agent reports all found IP addresses to server. Those addresses includes EC2 public and internal IPs.

To make solution more secure, you may create VPN connection from the build agent to your network. All you need is to setup build agent image that will connect to the network using VPN on machine start before running build agent.

0

Is there a way to tell the ec2 instance to start back up if a build configuration requests it?  I have this issue where if an ec2 instance is idle for say 30 minutes it shuts down and requires someone to manually start it back up.  It would be nice if it had the capability of starting it back up if a build configuration was assigned to the amazon cloud that it would start up the agent automatically.  This would be very ideal.

0

TeamCity checks the build queue for builds that may be started on a virtual machine. Once such build is queued, TeamCity will check and start an compatible EC2 image. Note, that TeamCity need to start every image once to fetch build agent parameters that are required to check compatibility later. Does it fits your needs?

0

Hi Eugene,

I think this is what I'm looking for, just so I'm following you correctly this is our scenerio:

BuildA is configured to run on two ec2 instances that are active (the agents are configured for specific projects not for all compatible projects, is there a way to keep these settings so that when ec2 shuts down it remembers the build configurations it was configured for?).  BuildA ran on both of these ec2 instances which have an idle time of 30 minutes.  After 30 minutes of no VCS submits the ec2 instances shutdown thus showing disconnected.  BuildA then has a checkin activating a build, the queue says "never" for when to run as there are no compatible agents.  Does teamcity then know from previous runs that it ran on ec2 instances thus those instances will start back up, thus becoming available for BuildA to run on?  Just want to understand the flow.

Thanks,
Jay

0

Yes, you are right. I'll rewrite the things more simply.

- Setup image for EC2 called, say, ami-555
- Create profile1 in TeamCity for ami-555 with long timeout
- Create profile2 in TeamCity for ami-555 (again) with ordinary timeout

When you see 'Never' in queue this mean only 'There is no connected agents to run the build'. Still, TeamCity could also start EC2 images while showing this in the UI. Cloud state is only show in 'Cloud' tab on 'Agents' page. There should be no need to start any of EC2 images explicitly.

0

Eugene,

What is the purpose of creating two profiles?  What are the advantages?  Can you tell me the behavior of teamcity and the agents when you set it up this way?

Thanks,
Jay

0

The only purpose of that is to make TeamCity have at least one EC2 agent running all the time.
Otherwise, it's enough to have one profile.

0

That is only the case if the profile you set to have a longer idle time is always building in the time frame?

What is the point of having the profile with a shorter idle time give you?  Maybe that is what is confusing me.

Thanks,
Jay

0

I'm confused about this as well, how would having 2 profiles solve this issue?

One profile would just have an agent on for a period of time, while the other profile would have agents shutdown after x amount of time. Does having that extra profile allow Teamcity to say 'i need more available resources(agents), let me start up more agents from this extra profile?'

0

In every profile under TeamCity you may specify a maximum number of agents that could be started to run builds. If there are some build in queue, TeamCity will start as much agents as needed to run those builds. This will be done automatically according to build compatibilities.

The extra account (with same EC2 credentials) will allow you to have 1 agent with timeout of X minutes and 10 agents with timeout Y minutes. If you set X = 30 hours, Y = 10 minutes you will have at least one EC2 agent running all the time, while extra build agents will be shut down after 10 minutes of idle time. This was my example.

Could you please re-describe your case, it seems there is something that I missed.

0

Is there a way to have the ec2 agents that were configured to start back up with the same TC configuration?  Such as "compatible build configurations".  If an ec2 agent is configured to only run for certain builds, when it shuts down due to idle time expiration, does it remember the settings when it comes back online?  Does it keep the compatible build configurations checked or does it default to run all configurations?

0

We plan to make explicit compatibility settings persistent for EC2 agents.

It will work If compatibility is set in terms of properties, i.e. in section "7. Agent Requirements" of a build configuration. You may register your own properties in build agent image in buildAgent.properties file. Note, EC2 agent have the following system properties:

ec2.ami-id   ami-*****
ec2.ami-launch-index   0
ec2.ami-manifest-path   path/
ec2.instance-id   i-***
ec2.instance-type   m1.small
ec2.local-hostname   ip-***.ec2.internal
ec2.local-ipv4   ***
ec2.public-hostname   ec2-****.compute-1.amazonaws.com
ec2.public-ipv4   ****
ec2.reservation-id   r-***


I believe those properties could be enough to setup compatibility of a build configuration to a build agent that was started from the EC2 image.

0

Thanks, we can do this to make sure the build configurations all run on ec2 instances.  I'm guessing that we would have to have all other builds that we don't want to run on ec2 instances to have agent requirements set to not run on those boxes.

Can you also answer mine and Josh's question about what does the second profile give you when setting it to a lesser idle time.  Why can't you just have one profile that has a longer idle time and thats it or were you trying to say that you set up one profile that has an agent that is always on then you have another profile that controls other instances that always turn off.  We still need to test that the shutdown instances come back up if no agents are available as when we first set this up I did not see this behavior.

0

To make configurations to running on EC2 you may add a requirement for it to not contain EC2 properties.
Please see http://confluence.jetbrains.net/display/TCD5/7.Agent+Requirements

Sorry, I tried to cover too complicated scenario. We may return to it later.

EC2 instances should be started by TeamCity automatically, once there are builds in the queue that are compatible with one of such instances. Please try it. Note, it may take up to 10-15 minutes for EC2 to start new machine. You TeamCity build agent EC2 image should be configured to autostart build agent on machine load. All EC2 information is shown in Cloud tab under Agents on server web UI.

BTW. Check image idle time and the number of allowed running EC2 instances.

0

Please sign in to leave a comment.