September 29 2010, 9:23 PM
Amazon’s EC2 service now allows you to boot from persistent EBS volumes—a boon to those of us who like to run the occasional server-hour but don’t want to mess with bundling AMIs and other atrocities. The AWS Management Console automates most tasks, but you might find yourself quickly running into a two-part annoyance: (1) if you use public AMIs, the size of your EBS volume is chosen for you; and (2) EBS volumes are not resizable—at least, not downwards. Since Amazon charges you for allotted space on EBS volumes, this “annoyance” can literally cost you tens of cents per month!
Luckily, there is a workaround, and it has the side benefit of allowing you to boot one EBS volume on different instance types—that is, boot up your volume on an “m1.small” instance one day, a “c1.medium” the next, and so on. What follows assumes familiarity with Amazon Web Services and EC2, UNIX/Linux, the command-line, computers, typing, pants-wearing, etc.
Ok: Launch an EC2 instance from an EBS-backed AMI. (I used the Canonical-provided Ubuntu 10.04 Lucid 32-bit EBS-backed AMI [ami-6c06f305].) This first instance is just a tool to help create our preferred environment and is not long for this world, so don’t get attached to it. For that reason, I launched an el-cheap-o “t1.micro” instance—or, I t-micro’d, as we like to say around here.
Once the instance boots up, a new EBS volume (15 GiB, in my case) is created and attached to your instance as /dev/sda1. Now create a blank EBS volume of your preferred size (I chose 2 GiB because I’m cheap) in the same availability zone and attach it to your instance. Specify a device mapping and make a note of it; I will assume /dev/sdf in the instructions below.
Connect to your instance:
ssh -i ~/.ssh/private.key ubuntu@ec2-x-x-x-x.compute-x.amazonaws.com
Format the newly attached volume:
sudo mkfs.ext3 /dev/sdf
Mount it:
sudo mkdir /mnt/new
sudo mount /dev/sdf /mnt/new
Copy everything over:
sudo rsync -ax --delete --progress / /mnt/new
While that copies, a quick note on IP addresses. Everytime you reboot or stop/start an instance, you are assigned new public and private IP addresses and hostnames. If your instance hosts public-facing services, that can make life difficult. Amazon’s solution is elastic IP addresses, which provide a static address for an instance or group of instances. This allows you to map a DNS A record to your instance or instances, have yourself a spiffy hostname, enjoy life, spend more time sailing, etc. Elastic IP addresses charge by bandwidth and have an hourly price when they are not attached to a running instance. Since I am cheap and leave my instances stopped as much as possible, employing one would cost me valuable pennies. Therefore, I created a startup script to update my DNS provider with my instance’s new IP address, and I also go ahead and update the hostname since my installation doesn’t seem to do that. Here’s my script, which I call from rc.local:
#!/bin/sh
META=http://169.254.169.254/latest/meta-data
HOSTNAME=`/usr/bin/curl -s $META/hostname | /bin/sed 's+\..*++g'`
hostname $HOSTNAME
echo $HOSTNAME > /etc/hostname
IPV4=`/usr/bin/curl -s $META/public-ipv4`
[DNS API MAGIC HERE]
You may have noticed the calls to http://169.254.169.254. This is an internal Amazon EC2 server that will report information about your instance—very helpful if you haven’t installed the EC2 API tools.
Now, this is the step of the walkthrough (there’s always at least one) where I gloss over crucially important facts and assume that you are smart enough to know all this beforehand even though you are a poor schmuck reading some tossed-off walkthrough on a blog: Depending on the AMI and instance type you chose, your fstab may contain entries for ephemeral storage and swap space. The t1.micro instance type, however, does not provide either. Therefore, if you want to be able to t-micro in the future, you will need to modify these fstab entries—most likely, /dev/sda[x>1]—to make them conditional. Otherwise, your el-cheap-o t1.micro will hang on boot as it waits around for storage devices that never show up.
Here are my highly exemplary edits, which add the nobootwait option to the ephemeral storage and swap mounts:
sudo nano /mnt/new/etc/fstab
proc /proc proc nodev,noexec,nosuid 0 0
/dev/sda1 / ext3 defaults 0 0
/dev/sda2 /mnt/eph auto defaults,nobootwait,comment=cloudconfig 0 0
/dev/sda3 none swap sw,nobootwait,comment=cloudconfig 0 0
That was easy! Unmount the new volume and exeunt:
sudo umount /mnt/new
exit
You now have a perfect copy on a smaller EBS volume, ready to swap in. Follow these steps to execute the swap, making sure to wait until each action is completely finished before proceeding to the next step.
- Stop (but do not terminate!) your instance.
- Detach both EBS volumes from your instance.
- Delete the larger, AMI-default EBS volume. (This is optional, I guess, but I told you not to get attached to it.)
- Reattach your new, smaller volume to your stopped instance and map it to
/dev/sda1. - Restart your instance.
Done! Now you have a twee little EC2 instance that you can stop and start on a whim. At this point, you could bundle your own private, EBS-backed AMI and deploy copies of it whenever you want, specifying the instance type on launch. However, my goal is to have one canonical volume that I can quickly boot on varying instance types. To do this, your options are two:
- While the instance is stopped, use the command-line EC2 API tools (specifically,
ec2-modify-instance-attribute) to change the instance type. Then restart the instance, either from the command line or via the Web console. - Boot instances of varying types from public AMIs, stop them, and detach and delete their default EBS volumes. Then attach your EBS volume and boot as needed.
Just remember that most instance types can boot either a 32-bit or 64-bit operating system, but not both. As of this writing, t1.micro is the only instance type that can boot both 32-bit and 64-bit.
So, let’s say, like me, you pick option two above. You’ve invested some time and about 75¢ in order to have t1.micro, m1.small, and c1.medium instances sitting idle and ready to boot your EBS volume. You probably would like a way to protect your instances from being accidentally terminated instead of stopped. While your EBS volume would be safe—since it was not created by any of your instances—it would be a waste of time and heaping fractions of dollars to set up another instance.
Luckily, you can use the EC2 API tools to lock your instance and protect it from your dumb, fat fingers:
ec2-modify-instance-attribute --disable-api-termination true [instance-id]
This only has to be done once and will survive repeated stop/starts. There’s no way to lock EBS volumes, but luckily they are harder to accidentally delete and you should be taking regular snapshots, anyway. (But not too many, moneybags—those snapshots are stored on S3 and added to your bill.)
Finally, a note on ephemeral storage: If you edited your fstab correctly, your ephemeral storage and swap should mount automatically when booting a non-t1.micro instance. If not, try mounting manually to see if you receive a helpful error:
sudo mount -a
If you are still unable to mount ephemeral storage, it’s possible that your original AMI didn’t map the devices, in which case you may want to start the instance manually and specify the mapping yourself. Also, it should be painfully obvious what happens to ephemeral storage when you stop or terminate an instance, but, if not, find a dictionary.