AWS EC2 boot issue and recovery guide

Recently, I’ve been doing quite a lot of Amazon Web Services stuff like EC2s and RDSs. Fortunately, we can now enjoy most of AWS services using the free tier account which is really a great opportunity provided by AWS. One of the most useful feature you can do with AWS is the ability to expand the size of your volumes without any lost of existing data. For those who aren’t familiar with AWS, a volume is basically a storage much like your Hard Disk on your Personal Computers.As I’ve mentioned earlier, the free tier is a good way to start, but there will come a time when you will eventually run out of space because the free tier only provide 5GB of storage. This happened to me so I was forced to expand my original 5GB of storage to 50GB and here’s how I did it.

1. Login to AWS console
2. Change the Shutdown Behavior of the instance you want to resize into Stop
3. Stop the instance
4. Create a snapshot of the volume you want to expand
5. Create a new volume out of the snapshot you just made, this is where the resizing happens
6. Detach the old volume from the instance you just stopped on step 3
7. Attach the new volume you just created
8. Start the instance

This should conclude the resizing process, but it wasn’t the case for me.
After Starting my instance, it didn’t successfully but returned an error on the System Log.
The error says:

VFS: Cannot open root device "LABEL=_/" or unknown-block(0,0)
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Now I’m smelling something fishy with our grub.conf configuration. But before we can go over and start messing things up with our config, remember that our instance isn’t starting so we can’t touch anything from there. Also, just to let you know that I’m using RHEL.Image
To get around with this, all we need is another instance where we can connect via ssh and mount our problematic instance’s volume to it. If you have another running instance where you can connect then that’ll save you more time but if you don’t have then you will have to launch a temporary instance.

If your second instance is ready, do the following:
1. Detach the volume attached to your problematic instance
2. Attach the volume you just detached to your temporary instance.
3. SSH to your new instance
4. Mount your new volume using:
sudo mount /dev/devicename /mnt
5. Edit grub.conf which should be at /mnt/boot/grub/grub.conf
6. Remove the first kernel stanza inside it and leave only “ec2-starter” which was provided by amazon by default.
7. Your grub.conf should look like this after you edit it, then save it.

default=0
timeout=5
splashimage=(hd0)/boot/grub/splash.xpm.gz
hiddenmenu
title rhel-server-x86_64-ec2-starter-6.4_20130130.0-11 (2.6.32-358.14.1.el6.x86_64)
   root (hd0)
   kernel /boot/vmlinuz-2.6.32-358.14.1.el6.x86_64 console=ttyS0 ro root=LABEL=_/
   initrd /boot/initramfs-2.6.32-358.14.1.el6.x86_64.img

Once you finished doing all of this, just detach the volume again and attach it to your problematic instance. Then reboot your instance. That should fix your booting issue with your ec2 instance.
If your issue persists, let me know by adding a comment below.

References:

http://amazonserver.blogspot.com/2013/01/recover-broken-amazon-ec2-instance.html

http://smoove-operator.blogspot.com/2014/01/red-hat-enterprise-linux-server-in.html

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: