EC2: very high number of API calls

Answered

We're currently evaluating running TeamCity up in EC2 but our ops guys just told me my TeamCity instance was generating "a very high number of API calls".  It was described to me as at least one per second.  That's bad!

From examples they sent me, it looks like calls from TC to DescribeImages.  I pulled down the TC cloud log file and attached it.  It appears as though the problems started on Jan 2nd when this was logged:

[2014-01-02 08:40:25,785]   WARN [ault'{id=cp1} 1] - on.async.AmazonAsyncClientImpl - Amazon connection error during Refresh Running Instances: Amazon EC2 connection (profile 'Windows default'{id=cp1}): An internal error has occurred Failed to fetch running instances

Shortly after that, there's plenty of "request limit exceeded" errors since I'm assuming that's when TC went wacko and AWS had to give it the smackdown.

Any assistance would be greatly appreciated.  This is definitely a show-stopped for us!



Attachment(s):
teamcity-clouds.log.zip
9 comments
Comment actions Permalink

What teamcity version are you using?

0
Comment actions Permalink

Are you still facing the same problem or didn't try again after this attempt?

0
Comment actions Permalink

This is using 8.0.5.  I haven't tried it again since shutting down the service last week.  I'm not sure I feel comfortable starting it up again until I can feel confident that it won't happen again.

0
Comment actions Permalink

Okay, thank you. There's no need to try again so far. I have enough information now to investigate.

0
Comment actions Permalink

Unfortunately, Amazon doesn't provide any exact information on what request rate is acceptable. It depends on the general size of your account, number of instances, images, their types, etc.

Currently, we don't throttle polling when Amazon replies with RequestLimitExceeded and that might potentially be a problem.

However, you can follow steps below to minimize the probability of this issue (we already have numerous customers, including ourselves - teamcity.jetbrains.com) that use amazon ec2 integration for production purposes:
1) reduce the number of EC2 cloud profiles. Teamcity makes separate Amazon API calls for every cloud profile.
2) I added a internal.property called "teamcity.amazon.data.sync.frequency.seconds" that you can use to manage sync frequency if you face the same problem again. To use the property you need to replace file <TeamcityInstallation>/ROOT/WEB-INF/plugins/cloud-amazon/server/cloud-amazon.jar with the attached one. Documentation can be found here: http://confluence.jetbrains.com/display/TCINT/Internal+%28debug%29+TeamCity+Properties+and+Abilities#Internal%28debug%29TeamCityPropertiesandAbilities-InstancesandImagesdatasyncfrequency.



Attachment(s):
cloud-amazon.jar
0
Comment actions Permalink

So, I only have one profile set up currently.  What is the default polling frequency?  Even once a second seems like overkill for us.  Especially in this situation where I was just testing and had only one project with no builds going on.  My cloud profile points to 3 stopped instances.  I'd like to use an AMI but we need to tag for billing which TC won't currently let me do.

0
Comment actions Permalink

Default polling frequency is 5 seconds and we are going to increase this value to 20 seconds in the nearest future as well as do some optimization in this area. Currently we poll Amazon with default frequency regardless of current teamcity state (# of running builds, etc.).

0
Comment actions Permalink

Due to the failure of AWS this evening in us-east-1, our TeamCity instance tried to start up instances on and on which always failed. After the services started to recover (and we moved cloud agents to another region), we started getting RequestLimitExceeded errors because TeamCity obviously tried to start to many instances within a short time of amount.

It's kinda annoying.

0
Comment actions Permalink

Hi Christian,

We have the related bug in our tracker: https://youtrack.jetbrains.com/issue/TW-34480, please vote for it. 

We are sorry for the inconvenience. 

0

Please sign in to leave a comment.