Unable to add new TeamCity EC2 cloud profile - "Failed to connect to Amazon EC2. Check your credentials"

TeamCity version: 9.0.1 (32116)

Summary of the below:

  • Access denied error is reported when attempting to connect to Amazon EC2
  • The credentials are definitely valid however (proof is below)
  • This prevents TeamCity from enumerating instances or starting instances to fulfil builds
  • No diagnostic information is logged as far as I can tell.


I'm not entirely sure how this has happened, but my TeamCity EC2 integration broke over Christmas. I came back today to discover that it would no longer start EC2 instances for new builds.

It reported a 401 error when attempting to communicate with Amazon. I tried resetting the access keys and adding them again, but this had no effect - when clicking "Check Connection / Fetch Parameter Values", the error message "Failed to connect to Amazon EC2. Check your credentials"

I tried deleting the profile and recreating it, but I was then unable to add a new one. I entered the same credentials, but clicking "Check Connection / Fetch Parameter Values", the same error message was shown.

I can't see anything in the log regarding this - the last time any of the TeamCity logs were updated was about 20 minutes ago.

I then thought that perhaps it might require some EC2 permissions that I didn't know about, so I tried some different IAM policy documents.

Firstly, I tried this:

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Action": "ec2:*",

      "Effect": "Allow",

      "Resource": "*"

    },

    {

      "Effect": "Allow",

      "Action": "elasticloadbalancing:*",

      "Resource": "*"

    },

    {

      "Effect": "Allow",

      "Action": "cloudwatch:*",

      "Resource": "*"

    },

    {

      "Effect": "Allow",

      "Action": "autoscaling:*",

      "Resource": "*"

    }

  ]

}

This had no effect; the error was the same. I then tried a complete all access administrator policy:

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": "*",

      "Resource": "*"

    }

  ]

}

Once again, this didn't work - the same error message was shown.

I then tested the credentials using PowerShell:

Set-AWSCredentials -SecretKey "xxxx" -AccessKey "yyyy"

Set-DefaultAWSRegion -Region eu-west-1

$(Get-EC2Instance | Measure-Object).Count

This listed the number of EC2 instances that I have in the eu-west-1 region, so I know the credentials are working correctly.

The default logging levels didn't write anything into the log.

I tried increasing the following logs to the DEBUG level:

  • jetbrains.buildServer.clouds.jetbrains.buildServer.serverSide.impl.FlushQueueVirtualAction
  • com.xerox.amazonws
  • jetbrains.buildServer.clouds


Unfortunately, no extra information was written into any log when I clicked the "Check Connection / Fetch Parameter Values" button so I'm not really sure how to debug this further. Seeing the individual messages that are sent back and forth between TeamCity and EC2 would be ideal.

I can work around this for now by manually turning the server on and off from the EC2 control panel but this isn't really ideal as I'm sure you can imagine. Strangely enough however, the instance does seem to shut down after the build - presumably this is handled by the agent and not by the TeamCity instance itself?

I would really appreciate getting this sorted out as soon as possible.

3 comments
Comment actions Permalink

The "Check Connection / Fetch Parameter Values" button  executes ec2-describe-regions (http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeRegions.html) against the US_EAST region (ec2.us-east-1.amazonaws.com)

Please try this query.

0
Comment actions Permalink

Any updates on this issue?

I hit the same on 9.0.4 (build 32407).

aws cli is able to describe regions

$ AWS_DEFAULT_REGION=us-east-1 AWS_ACCESS_KEY_ID=XXX AWS_SECRET_ACCESS_KEY=XXX aws ec2 describe-regions

{
    "Regions": [
        {
            "Endpoint": "ec2.eu-central-1.amazonaws.com",
            "RegionName": "eu-central-1"
....

0
Comment actions Permalink

Solved: make sure TC serever have time synced (ntpd)

0

Please sign in to leave a comment.