Automatically Delete EBS Volumes

I have EC2 cloud agents configured with my Team City instance. Unfortunately, this month I am being hit with a huge EC2 build because of the way TC is handling agents.  Specifically, when TC terminates a build agent, it doesn't delete the EBS volume associated with that build agent.  Consequently, the detached EBS volume is left "available" and accumulates storage fees.

How do I fix this?  Can TC be configured to delete the volumes?  Can I change the configuration of my AMI so that the volumes are not created in the first place?

Thanks,
-Scott

15 comments
Comment actions Permalink

Has nobody else encountered this?

0
Comment actions Permalink

Scott,

Seems like this is related to this feature request: TW-12517.
Or another way around: TW-16419.

Is fixing any of these would help in your case?

Unfortunately, none of them are avaialble currently.

Can you please detail why do you need to use EBS volumes?

0
Comment actions Permalink

Yep, same issue over here... I'am happy i found this as TC created 26volumes in just a few days... whould have been a hell of an awakening at the end of the month ( which it WILL be for a lot of other people )...

@JetBrains: You should really get on to this bug FAST, as it can cost a f****** fortune...

And: The issues in the other post are NOT related to this one from my point of view...

Also i don't see a workaround for this? How am i supposed to get rid of EBS usage?

0
Comment actions Permalink

I created a dedicated issue for your case:
http://youtrack.jetbrains.com/issue/TW-21137

Please share mode information there.

Unfortunately, due to http://youtrack.jetbrains.com/issue/TW-16419 the error could be expected.
Please vote for it

0
Comment actions Permalink

David,

Why would you want to use EBS instances instead of non-EBS ones?

0
Comment actions Permalink

I was creating the AMI when i was not aware that there is actual non-EBS storage ( And if you create an instance from a standard-AMI, amazon will not ask you ( obviously ) if EBS should be used ). Anyway, if EBS is not supported, at least the documentation should say so, because otherwise a lot of people will run into the same error.

0
Comment actions Permalink

I ran across this thread and have a related question.

We tried using EBS volumes (like the other posters) and hit the same issue as the other posters. We want to switch to S3-backed instances to solve the problem since it's clear that's what's expected. We have a Windows VHD but I can't find any instructions on creating an S3-backed instance from it. Can you point me to any documentation (or a tutorial) that explains how to create a S3-backed instance from a Windows VHD?

Thanks,

-Craig

P.S. - Note that I do know how to import my VHD to EC2. But, assuming the import is a required step, I don't see any instructions for how to use the imported VM to create a S3-backed instance.

0
Comment actions Permalink

Hello,

Thank you for the note.

I believe you may find any S3 based image to take as the base image, install there what you need and than rebundle it.
Another approach is described in the stackowerflow: thread http://stackoverflow.com/a/7037411/49811

0
Comment actions Permalink

Eugene,

 

I appreciate the reply. Unfortunately, it's not at all helpful. When launching Windows Server 2008 Instances on EC2, they're all EBS-backed.

 

Regarding the link you provided, the answer has a comment clearly indicating that the approach is only for Linux VM's. In fact, both answers are Linux only solutions.

 

Overall, I’m finding using TeamCity on EC2 extremely frustrating. On one hand, you guys claim you have support for TeamCity on EC2 with a Windows instance. But on the other hand, your "support" is so flawed that it cost me nearly $1,000 due to my volume unexpectedly continuing to run.

 

Frankly, it feels like the Windows scenario was never really tested considering there are no instructions on how to create a S3-backed Windows instance, and there's such an obvious bug (re: not deleting EBS volumes) that shouldn't have made it out the door, let alone remain unresolved. Finally, expecting users to manually terminate their volumes when a build agent terminates is a laughable suggestion. We have 50 people who want to use TC and none of them should need to understand the inner-workings of EC2, let alone be responsible for cleaning up after it. Particularly since TC should clean up after itself!

 

Unless 1 of these 2 questions is resolved, the only real option I see is to switch to a different CI solution:

 

1. How are users supposed to create S3-based Windows image in EC2? (so we avoid this bug with TC not deleting EBS volumes)

2. How soon will you guys finally fix this bug of EC2 volumes not being terminated when an agent is terminated?

 

Sincerely,

 

-Craig

0
Comment actions Permalink

Sorry, it was not clear to me if you run linux or windows.
I re-checked and seems there is no easy way to convert back to S3-backed AMI.

We run TeamCity.JetBrains.com (our public TeamCity installation) using EC2 support for years. All windows images are S3 backed.
Unfortunately, we does not support EBS backed images for now. We update the documentation to state it.

We plan to improve EC2 support (and finally implement support of EBS-backed instances) for 7.1.
As the work is done, the build will be available as 7.1. EAP and the relates issue(s) will be closed.
(please make sure you've vited for the issues to get notified on their status change)

0
Comment actions Permalink

Eugene,


Could you please provide step by step instructions on how to create a Windows S3 image? No one seems to be able to provide definitive steps for us; it would be a huge resource to your customers. And please don't point us to this link: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/creating-an-ami-s3.html. When I've tried creating a Windows Server 2008 instance for EC2, it's root device type is always EBS, despite what Amazon's instructions say.


Also, do you have any notion when 7.1 will be available?



-Craig
0
Comment actions Permalink

Eugene,

Any word on how to create a Windows S3 image and/or an estimate on when we can expect TeamCity 7.1?

Thanks,

-Craig

0
Comment actions Permalink

Hello,

I tried to run several tests for EBS images and found no leaked EBS Volumes not EBS Snapshots.
Could you please describe your images and the process of their start.
Does your images startup/activity deal with EBS images? Do you mount extra volumes to the image?

There are some EC2 docs describing how EBS volumes are treated:
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/Concepts_BootFromEBS.html#Instance_Termination
http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/AmazonEBS.html#DataPersistence

0
Comment actions Permalink

Guys,

A bit of a summary for the EBS issues in EC2 support:

There is a bug which occasionally can stop EBS-backed image instead of terminating it. It is filed as TW-12517.
Eugene has recently figured it out: it occurs only when the agent decides to stop itself before the server does so. (There is a logic to stop the agent instance both from the server and the agent on reaching an idle timout)
It only occurs sometimes and was not reproduced in our tests as usually server is more quick to process the timeout and correctly terminate the agent instance .

We will try to address the bug at least in TeamCity 7.1 (which should be released in first part of the Summer).


Currently, it is recommended to use non-EBS instances for TeamCity agent images. As there is no support for starting EBS images, EBS is no better as S3-backed ones.
If that is not possible, see also a workaround in the issue.

As to how to create S3-backed images: this is really a question for Amazon, not for us. It might be that this is hard or impossible in certailn cases. This does then mean that recommendation to use non-EBS images is hard to follow. We understand that and that contributes to the priority of the bug mentioned.

0
Comment actions Permalink

Had similar problem, had to change ebs settings for ami build, here is packer config:

 

"ami_block_device_mappings": [
{
"device_name": "/dev/sda1",
"volume_type": "gp2",
"volume_size": "400",
"delete_on_termination": true
}
]
0

Please sign in to leave a comment.