Troubles in US-East, migrating to US-West

I've been having increasing amounts of problems with my servers in the US-East-1 of Amazon Web Services. Fearful that this may be some fallout or aftershock of the Great EC2 Outage of 2011, I quickly sprung into action, migrating my servers over to US-West-1 (located in California).

It's important to note, however, that this had almost no impact on any of my production systems, as I already base my core services in two regions. These services continued to operate taking the full load of the system while us-east was having problems.

However, I took this as an opportunity to review my disaster recovery plan, and build up a new image that automatically updates from bitbucket repositories using an SSH key on reboot.

I launched my old instance, updated it quickly, and then used the following commands to bundle it from within the instance (as root):

# ec2-bundle-vol -c my_cert_file.pem -k my_key_file.pem -u MY_EC2_OWNER_ID -p my-ami-prefix -s 102400 -d /mnt

You can grab your EC2_OWNER_ID by using ElasticFox, double click on any instance you own, and copy the value for "Owner ID" which should be a number.

The next step is to upload this bundle to EC2:

ec2-upload-bundle -m /mnt/my-ami-prefix.manifest.xml -b my-ami-bucket -a MYACCESSKEYID -s MYSECRETACCESSKEY

Note that your bucket must be in the same region that you're bundling from. This means that if you want to actually migrate that manifest to another region, you have to use the ec2-migrate-manifest tool:

ec2-migrate-manifest -m /mnt/my-ami-prefix.manifest.xml \
   -c my_cert_file.pem \
   -k my_key_file.pem \
   -a MYACCESSKEYID \
   -s MYSECRETACCESSKEY \
   --region=us-west-1

And then you can re-upload it to a us-west-1 bucket using the ec2-upload-bundle command above. When you do this, you will be asked to make sure you want to upload to the other region, since it's not in the current region. Just say "y" for this since you've already migrated it using the ec2-migrate-manifest tool. This will take significantly longer since you're submitting cross-region


Lastly, you'll need to register this image with EC2 (or one copy in each region if you uploaded it to each region). I like to do this using boto:

>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region("us-west-1")
>>> conn.register_image(name="my-ami-name", image_location="my-ami-bucket/my-ami-name.manifest.xml")
u'ami-XXXXXX'

Then you're good to go, you'll get the AMI ID back that you can use to launch new instances in us-west.