AWS credentials error in connecting build agent with spot instances for jobs
Completed
Hey everyone, I'm hoping you can help me. I'm fairly new to assisting in the administration of TeamCity, and had a question about spot instance deploys. When a build agent is deployed to run a task via spot instance, it needs AWS credentials to successfully connect with it and proceed. This isn't happening however, and I need to now how teamcity interacts with spot instances. Is this an automated deployed triggered through an api command, or is there another built-in feature teamcity utilizes for this? I need to know so that I can correct/replace the incorrect key in question. The errors we're getting are below:
jetbrains.buildServer.clouds.amazon.error.AmazonCloudException: Amazon EC2 connection (profile 'newed-cloud-agent'{id=amazon-2}): Unable to load AWS credentials from any provider in the chain Failed to fetch running instances
at jetbrains.buildServer.clouds.amazon.sync.AmazonConnectionImpl.catchEC2Exception(AmazonConnectionImpl.java:149)
at jetbrains.buildServer.clouds.amazon.sync.SyncClientBase.catchEC2Exception(SyncClientBase.java:20)
at jetbrains.buildServer.clouds.amazon.sync.AmazonSyncClientImpl.getInstances(AmazonSyncClientImpl.java:181)
at jetbrains.buildServer.clouds.amazon.client.SynchronizeInstancesOperation.operation(SynchronizeInstancesOperation.java:34)
at jetbrains.buildServer.clouds.amazon.async.AmazonAsyncClientImpl$1.run(AmazonAsyncClientImpl.java:91)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:71)
at jetbrains.buildServer.util.ExceptionUtil$1.run(ExceptionUtil.java:41)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1164)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:15654)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:15630)
at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeSpotInstanceRequests(AmazonEC2Client.java:9316)
at com.amazonaws.services.ec2.AmazonEC2Client.describeSpotInstanceRequests(AmazonEC2Client.java:9292)
at jetbrains.buildServer.clouds.amazon.sync.AmazonSyncClientImpl$5.apply(AmazonSyncClientImpl.java:191)
at jetbrains.buildServer.clouds.amazon.sync.AmazonSyncClientImpl$5.apply(AmazonSyncClientImpl.java:181)
at jetbrains.buildServer.clouds.amazon.sync.AmazonConnectionImpl.catchEC2Exception(AmazonConnectionImpl.java:115)
Please sign in to leave a comment.
The minimal set of permissions you would need for your AWS account could be found at https://www.jetbrains.com/help/teamcity/setting-up-teamcity-for-amazon-ec2.html#SettingUpTeamCityforAmazonEC2-RequiredIAMpermissions.
Judging by the stacktrace you have provided, I would make a guess that instance list could not be retrieved - meaning that ec2:Describe* permission is unavailable. Could you please check that?
Please also feel free to reply with IAM description for the account in question.
Thanks so much for the insight. I'll take a look at the specific parts of the issue and bring back answers.
Hi there. I'll recap and provide updates here.
The cloud profile responsible for interacting with AWS to initiate the spot instance can't connect to the proper instance. Despite changing and replacing its keys twice, and ensuring that the associated AWS user has the highest permissions possible to execute this function, I am getting this error after running the Check connection/Fetch parameter values process:
Failed to connect to Amazon EC2. Unable to execute HTTP request: Connect to ec2.us-east-1.amazonaws.com:443 [ec2.us-east-1.amazonaws.com/54.239.28.168] failed: Connection timed out: connect
Security group rules look fine, as well. We've also been having issues with reaching teamcity both on the localhost and through the public web ui; there have been general issues with outbound internet/network connectivity from the instance itself lately as well, as of 2 April. These are circumstantial symptoms, but don't yet describe the whole. If you do have any further insight, I'd be hugely grateful.
https://teamcity-support.jetbrains.com/hc/en-us/community/posts/360007814580-AWS-credentials-error-in-connecting-build-agent-with-spot-instances-for-jobs
Is TeamCity never able to reach out the AWS side with this request? Is the host in question reachable from the TC server machine, and if it is, what are response times/packet loss rate? "Connection timed out: connect" usually hints on a connection issue, often due a firewall or proxy server in between (checking connection logs there if any are present could also be helpful).
Regarding the connectivity to the server itself, is that limited to the TeamCity instance or the whole server? Again, is there any correlation between connection quality while connected to TC and stats for pinging the server?
If above inquiries show nothing of interest, I would need teamcity-server.log and teamcity-clouds.log for further insight into what is happening on the server side. If you could change logging preset to "debug-cloud" as per this instruction (https://www.jetbrains.com/help/teamcity/teamcity-server-logs.html#TeamCityServerLogs-ChangingLoggingConfiguration) and do a few attempts to connect to spot instance / TC server itself, that would be particularly helpful.