Create AMI from existing EC2 instance.

There are times when we've configured an EC2 instance as per our requirements and we find need to create an AMI from the very instance. Recently I found this as a requirement while automating hadoop setup on EC2. We are using clouder-hadoop AMI for booting up hadoop cluster. However these machines need some additional softwares and we wanted to manage the setup using puppet.

So our requirement was to have an AMI with puppet installed and /etc/hosts containing entry corresponding to puppet master. We also wanted puppet to startup during bootup so that every software needed for our purpose was installed before machine was put to production use.

We followed following steps to get an AMI for our purpose:

1. Launch an EC2 instance using cloudera-hadoop AMI. Install puppet and configure /etc/hosts as per our requirement. We'll presume our instance id to be i-6666

2. Log off from the EC2 instance we just created.

3. On our machine we download ec2-ami-tools.zip and ec2-api-tools.zip

4. Create a directory ~/ec2

5. Extract ec2-ami-tools.zip and ec2-api-tools.zip to ~/ec2.

6. Copy EC2's PEM encoded X.509 certificate named something like cert-xxxxxxx.pem and PEM encoded RSA private key named something like pk-xxxxxxx.pem to ~/ec2. We can get these by logging to Amazon Web Services account at http://aws.amazon.com.

7. Setup following environment variables (I'm on Mac OS X, you might need to modify your bash_profile accordingly)

# Setup Amazon EC2 Command-Line Tools
export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/bin
export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem`
export EC2_CERT=`ls $EC2_HOME/cert-*.pem`
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/

8. Source the bash_profile. source ~/.bash_profile

9. Execute the command ec-bundle-instance. This command helps create an AMI from existing EC2 instance, bundle it and deploy it on S3. Following is an e.g. of the command in action:

ec2-bundle-instance i-6666 -b mybucket -p hadoopami -o -w

mybucket is name of S3 bucket and aws_access_key and aws_secret_key are Access Key ID and AWS Secret Access Key for the owner of the Amazon S3 bucket.

10. The above process takes some time and once done the EC2 instance will reboot. We can track the progress using

ec2-describe-bundle-tasks

11. Once image has been uploaded to S3 we need to register the image. We do so using ec2-register. Following is an e.g. of command in action:

ec2-register mybucket/hadoop-ami