AWS EC2 Cloud Support - Launching instances using IMDSv2
Hello, right now we're using the TeamCity Cloud Profile to launch agents in EC2.
I have been able to configure the profile and launch the agents successfully after following these guides. We have configured our AMIs to launch docker containers which are started automatically when the instance is first launched.
https://www.jetbrains.com/help/teamcity/agent-cloud-profile.html
https://www.jetbrains.com/help/teamcity/setting-up-teamcity-for-amazon-ec2.html
The problem I am having is configuring the instances to use the second version of instance metadata at launch. Having this configured is part one of the best practices for configuring EC2 instances.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html
If I configure the agents to use IMDSv2 after they are launched, then the agents are not able to read the meta data automatically since they are now required to supply http tokens. Since they fail to read the metadata, the agents appear to not know that they are part of the cloud profile, and thus are unable to successfully register to the main TeamCity server.
Does anyone know how I can configure this plugin to launch these instances with IMDSv2 configured, or perhaps of an alternative solution?
I'm not seeing any ability to do this in the settings. I tried setting this in the user script, but this breaks the agent as mentioned above. The agent will not be able to get the instance name.
Please sign in to leave a comment.
Hey Anatoly Cherenkov. I've just hit the same issue. If imdsv2 is required, an ec2 vm is spun up, but it never becomes available in TeamCity. Do you have a solution to this? I guess it should be a matter of using an up-to-date AWS SDK version. Not sure if any code changes would be needed. v1 and v2 are both available by default, so I'd expect the updated libraries to just use v2.
Required SDK/CLI versions https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/use-a-supported-sdk-version-for-imdsv2.html
Why this matters https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
Actually, upon closer inspection, the agent does connect to TeamCity, but doesn't become authorized. The agent's error logs is empty and the other logs don't seem to contain anything interesting. If I manually authorize the agent, it works.
The instance also doesn't get stopped/terminated after the build + timeout for termination. It just stays indefinitely and has to be stopped manually.
Hey, glad to see this post getting some attention again. We are still having this issue even after updating TeamCity to
TeamCity Professional 2022.10.2 (build 117025)
Alex Wilke well, for now it's just two users complaining on a >1 year old issue with no feedback from JetBrains, so I wouldn't get my hopes up just yet ;]
I'm doing some more testing and the node with the manually authorized agent was suddenly terminated in the middle of a build. So there's more issues around this.
We just started enforcing IMDSv2 for the entire account, so that's become a blocker for us.
It seems that when a build chain is triggered and a node is spawned and manually authorized, it will stay up until the end of the first build in the chain. Then it gets terminated and a new node is spawned, but is not usable for the next build in the chain due to "this build is waiting for the agent used for building its dependency".
Additionally, you might want to upvode the request to support IMDSv2 which is currently in the works. Thus you will get notified about the changes in the status of the issue: https://youtrack.jetbrains.com/issue/TW-75239
Hi Anatoly,
IMDSv1 should be disabled. Enabling v1 alongside v2 undoes any benefits of enabling v2.
Until TW-75239 is implemented, TeamCity is forcing a less secure setup.
> In a retrospective on public cloud breaches, Christophe Tafani-Dereeper, Rami
> McCarthy and Houston Hopkins describe the exploitation of an SSRF
> vulnerability to steal application cloud credentials as one of the four main
> causes of cloud security incidents in 2022.
IMDSv2 was introduced in 2019.
Thanks
Looks like this is getting fixed! (2022.10.04)
https://youtrack.jetbrains.com/issue/TW-75239
Indeed. The updated plugin is available for download. I've already installed it, but haven't had the time to test yet.