Amazon EC2 Deployment with Boto

Here is a quick guide to firing up new servers using boto, a python interface to Amazon Web Services. Hook this in with your automated install of your application using fabric and you are well on your way to web-scale.

First we create an EC2Connection class that will house all of our functions, including a connect() function

from boto.ec2.connection import EC2Connection

class EC2Conn:

        def __init__(self):
                self.conn = None
                self.access_key = 'my-amazon-access-key'
                self.secret_key = 'my-amazon-secret-key'


        def connect(self):
                self.conn = EC2Connection(self.access_key, self.secret_key)

Then lets decide the type of instance we want to create. You can get a list of AMI instances on their website, or search for them within their own deployment GUI. We are going to deploy a micro instance running Amazon Linux. The AMI Id for this instance is ami-aecd60c7 and the instance size string is t1.micro (other instance sizes here.)

In order to access the instance via SSH you will want assign a keypair and also define a security group that opens traffic on port 22 for SSH, 80/443 for web requests and anything else you plan to do. You can create keypairs and define security groups programatically but since it's often a one-time thing I just went and did it via the deployment GUI in amazon.

WEB : { 'image_id' : 'ami-aecd60c7',
        'instance_type' : 't1.micro',
        'security_groups' : [ 'my-security-group-name' ],
        'key_name' : 'my-keypair-name',
      }

Now we are ready to deploy an instance. With Amazon, you first create an instance reservation which Amazon will then provision for you. You can actually request several instances to be created at once in a single reservation, but the default is 1. Naturally, it takes a few minutes for the instance to boot up (depending on size, region selected, etc) so there is a "state" variable you continually check to find out when it is ready.

Create a reservation and wait until instance is ready.

# Key names are the same as the keyword arguments required by boto
WEB : {
        'image_id' : 'ami-aecd60c7',
        'instance_type' : 't1.micro',
        'security_groups' : [ 'my-security-group-name' ],
        'key_name' : 'my-keypair-name',
      }


def create_instance(self):

        reservation = self.conn.run_instances( **WEB)
        instance = reservation.instances[0]

        while instance.state != 'running':
                time.sleep(5)
                instance.update() # Updates Instance metadata
                print "Instance state: %s" % (instance.state)

        print "instance %s done!" % (instance.id)
        print "instance IP is %s" % (instance.ip_address)
        print "You can now SSH into this server with ec2-user@%s" % (instance.ip_address)

Just like that you have an Amazon instance you can mess around with.

Maybe you want to assign an Elastic IP Address that you have already provisioned?

def link_instance_and_ip(self, instance_id, ip):
        success = self.conn.associate_address(instance_id=instance_id,
                                              public_ip=ip)
        if success:
                print "Sleeping for 60 seconds to let IP attach"
                time.sleep(60)

        return success


success = self.link_instance_and_ip(instance.id, address)
instance.update() # Since there is a  new IP

Maybe you want to update your Fabric installation scripts to deploy a server first and then install the necessary packages?

def deploy_web():

        # Create New Instance
        instance = create_new_instance()

        # Set Env Host String
        env.host_string = "ec2-user@%s" % (instance.ip_address)

        # Begin Installation

        user = 'cmon_son'
        remote_home_dir = '/home/' + user

        with settings(warn_only=True):
                sudo('useradd -U -m %s' % user)

        # Install packages with yum
        sudo('yum install -y %s' % (" ".join(PACKAGES_LIST)) )

        # Install pip
        sudo('curl -O http://pypi.python.org/packages/source/p/pip/pip-1.0.tar.gz')
        run('tar xvfz pip-1.0.tar.gz')
        sudo('cd pip-1.0 && python setup.py install')

        # Install virtualenv
        sudo('pip install virtualenv')
        venv_name = '%s-env' % user
        venv = os.path.join(remote_home_dir, venv_name)
        sudo('virtualenv --no-site-packages %s' % venv)

        # Install python requirements
        put('requirements.txt', remote_home_dir, use_sudo=True)
        sudo('%s/bin/pip install -r %s/requirements.txt' % (venv, remote_home_dir))

        deploy_the_codes(HOME_DIR, remote_home_dir)

Full deploy code here.

from boto.ec2.connection import EC2Connection

import sys
import time

MAIN_IP = '10.81.18.01'
MAIN_SG = 'some-security-group-name'
MAIN_KP = 'some-kp-name'

SERVER_TYPES = {
                'web' : {
                       'image_id' : 'ami-aecd60c7',
                       'instance_type' : 't1.micro',
                       'security_groups' : [MAIN_SG],
                       'key_name' : MAIN_KP,
                        },
}


class EC2Conn:

        def __init__(self):
                self.conn = None
                self.access_key = 'my-amazon-access-key'
                self.secret_key = 'my-amazon-secret-key'


        def connect(self):
                self.conn = EC2Connection(self.access_key, self.secret_key)

        def create_instance(self, instance_type='web', address=None):
                reservation = self.conn.run_instances( **SERVER_TYPES[instance_type])
                print reservation
                instance = reservation.instances[0]
                time.sleep(10)
                while instance.state != 'running':
                        time.sleep(5)
                        instance.update()
                        print "Instance state: %s" % (instance.state)

                print "instance %s done!" % (instance.id)

                if address:
                        success = self.link_instance_and_ip(instance.id, address)
                        if success:
                                print "Linked %s to %s" % (instance.id, address)
                        else:
                                print "Falied to link%s to %s" % (instance.id, address)
                        instance.update()

                return instance

        def link_instance_and_ip(self, instance_id, ip=MAIN_IP):
                success = self.conn.associate_address(instance_id=instance_id,
                                                                        public_ip=ip)
                if success:
                        print "Sleeping for 60 seconds to let IP attach"
                        time.sleep(60)

                return success

        def unlink_instance_and_ip(self, instance_id, ip=MAIN_IP):
                return self.conn.disassociate_address(instance_id=instance_id,
                                                                        public_ip=ip)

        def get_instances(self):
                return self.conn.get_all_instances()


def create_new_instance(address=MAIN_IP):
        a = EC2Conn()
        a.connect()
        return a.create_instance(address=address)

Afterword

I am currently working on a side project where the primary goal is to explore many of the widely-used open source alternatives to some of the pieces we attempted to build in-house at my last job: a werkzeug-powered app framework, a job queue, a local cache layer. But also to explore some of the other technologies that have become popular over the past few years: NoSQL, Node.js and who knows what else.

Posts (so far) in this Series

Engineer Without a Cause

Amazon EC2 Deployment with Boto

Comments !

About

Comments !

About

Social