Installation Automation: Fabric Basics

Automated Installation and Deployment

One of the smartest things we ever did in dev at my last job was to hard code and formalize the installation and deployment process of our application. To me, who was not only ignorant of deploy methodologies but had also inherited a 10-page perpetually out of date wiki document describing the installation, this seemed like an extremely daunting task. Now I can't imagine it any other way and so will be formalizing this from the start. Fabric was a big part of this and allowed us to execute all of our installation commands on the remote server.

Fabric

First install fabric with pip

$ pip install fabric

Then we will create a very simple fabric command to test using fabric's run command

from fabric.api import run

def what_is_my_name():
    run('whoami')

We will now run this command on our localhost

$ fab -H localhost what_is_my_name
[localhost] Executing task 'what_is_my_name'
[localhost] run: whoami
[localhost] out: jhull

You can also run commands as a superuser using the sudo command

from fabric.api import run, sudo

def what_is_my_name():
    run('whoami')

def what_is_sudos_name():
    sudo('whoami')

When run on our localhost

$ fab -H localhost what_is_sudos_name

[localhost] Executing task 'what_is_sudos_name'
[localhost] sudo: whoami
[localhost] out: sudo password:
[localhost] out: root

Lets list the fabric commands we have:

$ fab -l
Available commands:

    what_is_my_name
    what_is_sudos_name

Now lets run these commands on two different hosts.

$ fab -H localhost,jh@78.123.24.65 what_is_my_name
[localhost] Executing task 'what_is_my_name'
[localhost] run: whoami
[localhost] out: jhull

[78.123.24.65] Executing task 'what_is_my_name'
[78.123.24.65] run: whoami
[78.123.24.65] Login password for 'jh':
[78.123.24.65] out: jh

Done.
Disconnecting from 78.123.24.65... done.
Disconnecting from localhost... done.

A common case is that you have a fixed set of hosts that you will want to batch commands on. Might be for code deployment, backups or any server updates in general. Lets add shortcuts to these servers. At the same time we will add our SSH key so that we won't have to enter passwords each time.

from fabric.api import env

HOSTS = {
   'web1' : { 'user': 'dev',
              'host': '84.156.12.134'
              },
   'web2' : { 'user': 'dev',
              'host': '84.156.45.24'
              },
   }

# Set SSH Key
env.key_filename = '/path/to/.ssh/key'

def host(hostname):
        env.hosts.append("%s@%s" % ( HOSTS[hostname]['user'],
                                     HOSTS[hostname]['host']))

Now we can execute any fabric command

$ fab host:web1 what_is_my_name
[84.156.12.134] Executing task 'what_is_my_name'
[84.156.12.134] run: whoami
[84.156.12.134] out: dev

Done.
Disconnecting from 84.156.12.134... done.

What was most important piece for us was the actual installation and server setup. Our deployment code ended up being close to 2000 lines of fabric commands and code. It allowed us to spin up any instance, configure mysql, nginx, supervisor, redis and a million other things in all under 5 minutes consistently and efficiently. This was huge and allowed us to very recover from common EC2 outages with great agility and without stress.

Possible fabric commands that might come in handy

def install_nginx():
    """ Install nginx and copy over our config file """
    sudo('yum install -y nginx')
    put('nginx.conf', '/etc/nginx/')
    sudo('service nginx start')


def update_code(path='/my/app/directory')
    """Update code with git by pulling
    latest production branch from origin"""

    with cd(path):
        run('git pull origin production')


def update_code_svn(path='/my/app/directory')
    """Update code with svn """

    with cd(path):
        run('svn up')


def server_updates():
    """ Run yum update """
    sudo('yum update -y')


def run_tests():
    """ Run test suites"""
    run('nosetests --with-coverage my_test_suite.py')

Afterword

I am currently working on a side project where the primary goal is to explore many of the widely-used open source alternatives to some of the pieces we attempted to build in-house at my last job: a werkzeug-powered app framework, a job queue, a local cache layer. But also to explore some of the other technologies that have become popular over the past few years: NoSQL, Node.js and who knows what else.

Posts (so far) in this Series

Comments !

About

Started writing one year ago, the day after heading out to travel around the world for a year without a cause.
Current Location: New York, New York

Previously:
Mexico City, Mexico
Tokyo, Japan
Hanoi, Vietnam
Vientiane, Laos
Phuket, Thailand
Kathmandu, Nepal
Rajastan, India
Kerala, India
Mumbai, India
Freetown, Sierra Leone
Koidu, Sierra Leone
Mombasa, Kenya
Nairobi, Kenya
Kigali, Rwanda
Rwinkwavu, Rwanda
Boston, MA

Latest Posts

Port Forward an old Airport Express

If Developers Took Steroids

We Need Elon Musk

Crossword Scraper

Git Conflicts in your Binary Files

Japanese Sidewalk Interfaces

Introducing Kickbacker

Real Life: Google Glass Done Wrong

Tux Trashcans

How Angry are your Developers?

A 500 Startups Model for the Art World

Unsubscribe from Black Friday/Cyber Monday

Copyrighting Art into Obscurity

Crack WiFi Passwords with aircrack

Using Sandy as an Excuse to Email Spam Customers