Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Bootstrap your DO infrastructure unsing Ansible without dynamic inventory (version for Ansible v2.0+ and DO API v2.0)
#!/bin/bash
#
# What is that
# ============
#
# This script will help you setting up your digital ocean
# infrastructure with Ansible v2.0+ and DO API v2
#
# Usually, when working with DO, one is supposed to use digital_ocean.py
# inventory file, and spin up instances in a playbook.
# However, this approach is very inconvenient for several reasons:
# - it is veeeery slow
# - your droplets won't have "ansible" names, just IPs
# - ... consequently, putting them into groups is a pain (but doable)
# - but if you succeed doing so, groups will only be available at run time
# - you are required to have 'localhost' in your inventory, which basically ruins 'all' group purpose
# - etc...
# All in all, it is barely usable.
#
# This script attempts to make things easier. It works the following way:
#
# - it will read 'hosts' file in an inventory directory (passed as the first argument)
# - it will spin up a DO droplet for each of these hosts (in parallel !) using ansible
# (note that you can specify image type, droplet size, etc... in the inventory itself)
# - it will generate an complementary inventory file (in the inventory
# directory) containing droplets names along with their IP addresses, so you
# won't need hitting the DO API anymore when running ansible.
# Using this script, you can work like you use to do with bare metal machines,
# without the chicken and egg problem of network configuration)
# The script itself can:
# - spin up droplets (`do_boot2.sh inventory_directory_path`)
# - destroy droplets (`do_boot2.sh inventory_directory_path deleted`)
# Since the droplets are created in parallel, you will also save tons of time
# when building an infrastructure involving a bunch of droplets. As an example,
# creating 8 droplets takes 130 seconds using this script, almost 570 using a
# "classic approach". Destroying them takes 10 secs with this script, 55 using
# a classic approach. So in the end, you get a 5 fold speed up, and a much
# better usability. Of course, the more droplets, the more gain.
#
# DO_API_TOKEN environment variable must be set.
#
# Change defaults below
# ---------------------
# Digital Ocean default values
# You can override them using do_something in your inventory file
# Example:
#
# [www]
# www1 do_size_slug="1gb" do_region_slug="nyc1" do_image=12345
# ...
#
# If you don't override in your inventory, the defaults below will apply
DEFAULT_SIZE="512mb" # 512mb (override with do_size_slug)
DEFAULT_REGION="ams2" # ams2 (override with do_region_slug)
DEFAULT_IMAGE="ubuntu-14-04-x64" # Ubuntu 14.04 x64 (override with do_image_slug)
DEFAULT_KEY=785648 # SSH key, change this ! (override with do_key)
# localhost entry for temporary inventory
# This is a temp inventory generated to start the DO droplets
# You might want to change ansible_python_interpreter
LOCALHOST_ENTRY="localhost ansible_python_interpreter=/usr/bin/python2"
# Set state to present by default
STATE=${2:-"present"}
# digital_ocean module command to use
# name, size, region, image and key will be filled automatically
COMMAND="state=$STATE command=droplet private_networking=yes unique_name=yes"
# ---------------------
function bail_out {
echo -e "\033[0;31m"
echo $1
echo -e "\033[0m"
echo -e "Usage: $0 <inventory_directory> [present|deleted]\n"
echo -e "\tinventory_directory: the directory containing the inventory goal (compulsory)"
echo -e "\tpresent: the droplet will be created if it doesn't exist (default)"
echo -e "\tdeleted: the droplet will be destroyed if it exists\n"
exit 1
}
# Check that inventory is a directory
# We need this since we generate a complementary inventory with IP addresses for hosts
INVENTORY=$1
[[ ! -d "$INVENTORY" ]] && bail_out "Inventory does not exist, is not a directory, or is not set"
[[ -z "$DO_API_TOKEN" ]] && bail_out "DO_API_TOKEN not set. Please visit https://cloud.digitalocean.com/settings/applications"
JQ=`which jq` || bail_out "Unable to find required binary 'jq'. Please install it first (http://stedolan.github.io/jq/)"
# Get a list of hosts from inventory dir
HOSTS=$(ansible -i $1 --list-hosts all | awk '{ print $1 }' | tr '\n' ' ')
# Clean up previously generated inventory
rm ${INVENTORY}/generated > /dev/null 2>&1
# Creating temporary inventory with only localhost in it
TEMP_INVENTORY=$(mktemp)
echo Creating temporary inventory in ${TEMP_INVENTORY}
echo ${LOCALHOST} > ${TEMP_INVENTORY}
# Create droplets in //
for i in ${HOSTS}; do
SIZE=$(grep $i $1/hosts | grep do_size_slug | sed -e 's/.*do_size_slug=\(\d*\)/\1/')
REGION=$(grep $i $1/hosts | grep do_region_slug | sed -e 's/.*do_region_slug=\(\d*\)/\1/')
IMAGE=$(grep $i $1/hosts | grep do_image_slug | sed -e 's/.*do_image_slug=\(\d*\)/\1/')
KEY=$(grep $i $1/hosts | grep do_key | sed -e 's/.*do_key=\(\d*\)/\1/')
SIZE=${SIZE:-$DEFAULT_SIZE}
REGION=${REGION:-$DEFAULT_REGION}
IMAGE=${IMAGE:-$DEFAULT_IMAGE}
KEY=${KEY:-$DEFAULT_KEY}
if [ "${STATE}" == "present" ]; then
echo "Creating $i of size $SIZE using image $IMAGE in region $REGION with key $KEY"
else
echo "Deleting $i"
fi
# echo " => $COMMAND name=$i size_id=$SIZE image_id=$IMAGE region_id=$REGION ssh_key_ids=$KEY"
ansible localhost -c local -i ${TEMP_INVENTORY} -m digital_ocean \
-a "$COMMAND name=$i size_id=$SIZE image_id=$IMAGE region_id=$REGION ssh_key_ids=$KEY" &
done
wait
# Now do it again to fill up complementary inventory
if [ "${STATE}" == "present" ]; then
for i in ${HOSTS}; do
echo Checking droplet $i
IP=$(ansible localhost -c local -i $TEMP_INVENTORY -m digital_ocean -a "state=present command=droplet unique_name=yes name=$i" | sed -e 's/localhost | success >> //' | $JQ '.droplet.networks.v4[] | select(.type == "public") | .ip_address' | cut -f2 -d'"')
echo "$i ansible_ssh_host=$IP" >> ${INVENTORY}/generated
done
fi
echo "All done !"
@jtktam

This comment has been minimized.

Show comment Hide comment
@jtktam

jtktam May 11, 2015

hi. i am trying to figure out how to best use your script. I got it running and it output a file called "generated"

how do i use this file in my ansible?

thanks

jtktam commented May 11, 2015

hi. i am trying to figure out how to best use your script. I got it running and it output a file called "generated"

how do i use this file in my ansible?

thanks

@jtktam

This comment has been minimized.

Show comment Hide comment
@jtktam

jtktam Jun 2, 2015

seems like the latest ansible update has broken the last part of this script, it doesn't createthe "generated" file any more

parse error: Invalid numeric literal at line 1, column 10
All done !

jtktam commented Jun 2, 2015

seems like the latest ansible update has broken the last part of this script, it doesn't createthe "generated" file any more

parse error: Invalid numeric literal at line 1, column 10
All done !

@jtktam

This comment has been minimized.

Show comment Hide comment
@jtktam

jtktam Jun 2, 2015

if you reading these comments, I have fixed the problem in my fork

jtktam commented Jun 2, 2015

if you reading these comments, I have fixed the problem in my fork

@lbischof

This comment has been minimized.

Show comment Hide comment
@lbischof

lbischof Jun 20, 2015

Thank you jtktam!

Thank you jtktam!

@ericln

This comment has been minimized.

Show comment Hide comment
@ericln

ericln Oct 8, 2015

parsing of the override is incorrect

ericln commented Oct 8, 2015

parsing of the override is incorrect

@apinnecke

This comment has been minimized.

Show comment Hide comment
@apinnecke

apinnecke Nov 15, 2015

With latest Versions of dopy and ansible the parameter api_token is missing. Fixed in my fork. ;)

With latest Versions of dopy and ansible the parameter api_token is missing. Fixed in my fork. ;)

@aatchison

This comment has been minimized.

Show comment Hide comment
@aatchison

aatchison Jan 6, 2017

I have to say the blog post was right on, and this script was exactly what I was looking for and couldn't find. Generating a host file is much more compatible with my existing playbooks. I thank you. Also you fork-ers, you helped too.
Cheers,
Arron

I have to say the blog post was right on, and this script was exactly what I was looking for and couldn't find. Generating a host file is much more compatible with my existing playbooks. I thank you. Also you fork-ers, you helped too.
Cheers,
Arron

@Routhinator

This comment has been minimized.

Show comment Hide comment
@Routhinator

Routhinator Apr 15, 2017

Ah yeah, there it is. The output of --list-hosts all has changed to:

  hosts (3):
    host1
    host2
    host3

So awk output ends up being:

hosts host1 host2 host3

Change Line 94 of jtktams fork to:

HOSTS=$(ansible -i $1 --list-hosts all | awk '{ print $1 }' | awk '{if (NR!=1) {print}}' | tr '\n' ' ')

Routhinator commented Apr 15, 2017

Ah yeah, there it is. The output of --list-hosts all has changed to:

  hosts (3):
    host1
    host2
    host3

So awk output ends up being:

hosts host1 host2 host3

Change Line 94 of jtktams fork to:

HOSTS=$(ansible -i $1 --list-hosts all | awk '{ print $1 }' | awk '{if (NR!=1) {print}}' | tr '\n' ' ')
@Routhinator

This comment has been minimized.

Show comment Hide comment
@Routhinator

Routhinator Apr 16, 2017

Additionally the parsing does not work if there are more than one override:

Config example:

host1 do_region_slug="nyc1" do_size_slug="512mb"

Current line example:

SIZE=$(grep $i $1/hosts | grep do_size_slug | sed -e 's/.*do_size_slug=\(\d*\)/\1/'

Resulting value of $SIZE:

"nyc1" do_size_slug="512mb"

Need to remove everything in the line after the second "

This is fixed in my fork.

Working lines 106-109

  SIZE=$(grep $i $1/hosts | grep do_size_slug | sed -e 's/.*do_size_slug=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')
  REGION=$(grep $i $1/hosts | grep do_region_slug | sed -e 's/.*do_region_slug=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')
  IMAGE=$(grep $i $1/hosts | grep do_image_slug | sed -e 's/.*do_image_slug=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')
  KEY=$(grep $i $1/hosts | grep do_key | sed -e 's/.*do_key=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')

Additionally the parsing does not work if there are more than one override:

Config example:

host1 do_region_slug="nyc1" do_size_slug="512mb"

Current line example:

SIZE=$(grep $i $1/hosts | grep do_size_slug | sed -e 's/.*do_size_slug=\(\d*\)/\1/'

Resulting value of $SIZE:

"nyc1" do_size_slug="512mb"

Need to remove everything in the line after the second "

This is fixed in my fork.

Working lines 106-109

  SIZE=$(grep $i $1/hosts | grep do_size_slug | sed -e 's/.*do_size_slug=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')
  REGION=$(grep $i $1/hosts | grep do_region_slug | sed -e 's/.*do_region_slug=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')
  IMAGE=$(grep $i $1/hosts | grep do_image_slug | sed -e 's/.*do_image_slug=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')
  KEY=$(grep $i $1/hosts | grep do_key | sed -e 's/.*do_key=\(\d*\)/\1/' | sed -r 's/(([^"]*"){2}).*/\1/')
@Patrick-V-Wirth

This comment has been minimized.

Show comment Hide comment
@Patrick-V-Wirth

Patrick-V-Wirth Apr 20, 2018

LOCALHOST_ENTRY="localhost ansible_python_interpreter=/usr/bin/python2"

This variable is not used at line 102
echo ${LOCALHOST} > ${TEMP_INVENTORY}

should be
echo ${LOCALHOST_ENTRY} > ${TEMP_INVENTORY}

LOCALHOST_ENTRY="localhost ansible_python_interpreter=/usr/bin/python2"

This variable is not used at line 102
echo ${LOCALHOST} > ${TEMP_INVENTORY}

should be
echo ${LOCALHOST_ENTRY} > ${TEMP_INVENTORY}

@Patrick-V-Wirth

This comment has been minimized.

Show comment Hide comment
@Patrick-V-Wirth

Patrick-V-Wirth Apr 20, 2018

DEFAULT_KEY is the Digital Ocean ssh key Fingerprint. That took me a few minutes to figure out.
It can be generated as seen on Digital Ocean with the following command:

ssh-keygen -lv -E md5 -f ./id_rsa.pub

DEFAULT_KEY is the Digital Ocean ssh key Fingerprint. That took me a few minutes to figure out.
It can be generated as seen on Digital Ocean with the following command:

ssh-keygen -lv -E md5 -f ./id_rsa.pub

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment