Skip to content

Instantly share code, notes, and snippets.

@karmi
Created March 16, 2012 16:09
Show Gist options
  • Save karmi/2050769 to your computer and use it in GitHub Desktop.
Save karmi/2050769 to your computer and use it in GitHub Desktop.
Bootstrap, install and configure ElasticSearch with Chef Solo
.DS_Store
Gemfile.lock
*.pem
node.json
tmp/*
!tmp/.gitignore

Bootstrap, install and configure ElasticSearch with Chef Solo

The code in this repository bootstraps and configures a fully managed Elasticsearch installation on a EC2 instance with EBS-based local persistence.

Download or clone the files in this gist:

curl -# -L -k https://gist.github.com/2050769/download | tar xz --strip 1 -C .

First, in the downloaded node.json file, replace the access_key and secret_key values with proper AWS credentials.

Second, create a dedicated security group in the AWS console for ElasticSearch nodes. We will be using group named elasticsearch-test.

Make sure the security groups allows connections on following ports:

  • Port 22 for SSH is open for external access (the default 0.0.0.0/0)
  • Port 8080 for the Nginx proxy is open for external access (the default 0.0.0.0/0)
  • Port 9300 for in-cluster communication is open to the same security group (use the Group ID for this group, available on the "Details" tab, such as sg-1a23bcd)

Third, launch a new instance in the AWS console:

  • Use a meaningful name for the instance. We will use elasticsearch-test-chef-1.
  • Create a new "Key Pair" for the instance, and download it. We will be using a key named elasticsearch-test.
  • Use the Amazon Linux AMI (ami-1b814f72). Amazon Linux comes with Ruby and Java pre-installed.
  • Use the m1.large instance type. You may use the small or even micro instance type, but the process will take very long, due to AWS constraints (could be hours instead of minutes).
  • Use the security group created in the first step (elasticsearch-test).

Copy the SSH key downloaded from AWS console to the tmp/ directory of this project and change its permissions:

cp ~/Downloads/elasticsearch-test.pem ./tmp
chmod 600 ./tmp/elasticsearch-test.pem

Once the instance is ready, copy its "Public DNS" in the AWS console (eg. ec2-123-40-123-50.compute-1.amazonaws.com).

We can begin the "bootstrap and install" process now.

Let's setup the connection details, first:

HOST=<REPLACE WITH YOUR PUBLIC DNS>
SSH_OPTIONS="-o User=ec2-user -o IdentityFile=./tmp/elasticsearch-test.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

Let's copy the files to the machine:

scp $SSH_OPTIONS bootstrap.sh patches.sh node.json solo.rb $HOST:/tmp

Let's bootstrap the machine (ie. install neccessary packages, download cookbooks, etc):

time ssh -t $SSH_OPTIONS $HOST "sudo bash /tmp/bootstrap.sh"
time ssh -t $SSH_OPTIONS $HOST "sudo bash /tmp/patches.sh"

Let's launch the Chef run with the chef-solo command to provision the system:

time ssh -t $SSH_OPTIONS $HOST "sudo chef-solo -N elasticsearch-test-1 -j /tmp/node.json"

Once the Chef run successfully finishes, you can check whether ElasticSearch is running on the machine (leave couple of seconds for ElasticSearch to have a chance to start...):

ssh -t $SSH_OPTIONS $HOST "curl localhost:9200/_cluster/health?pretty"

You can also connect to the Nginx-based proxy:

curl http://USERNAME:PASSWORD@$HOST:8080

And use it for indexing some data:

curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/document/1" -d '{"title" : "Test 1"}'
curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/document/2" -d '{"title" : "Test 2"}'
curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/document/3" -d '{"title" : "Test 3"}'
curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/_refresh"

Or performing searches:

curl "http://USERNAME:PASSWORD@$HOST:8080/_search?pretty"

You can also use the provided service to check ElasticSearch status:

ssh -t $SSH_OPTIONS $HOST "sudo service elasticsearch status -v"

Of course, you can check the ElasticSearch status with Monit:

ssh -t $SSH_OPTIONS $HOST "sudo monit reload && sudo monit status -v"

(If the Monit daemon is not running, start it with sudo service monit start first. Notice the daemon has a startup delay of 2 minutes by default.)

The provisioning scripts will configure the following on the target instance:

  • Install Nginx and Monit
  • Install and configure Elasticsearch via the cookbook
  • Create, attach, format and mount a new EBS disk
  • Configure Nginx as a reverse proxy for Elasticsearch with HTTP authentication
  • Configure Monit to check Elasticsearch process status and cluster health

This repository comes with a collection of Rake tasks which automatically create the server in Amazon EC2, and perform all the provisioning steps. Install the required Rubygems with bundle install and run:

time bundle exec rake create NAME=elasticsearch-test-from-cli

http://www.elasticsearch.org/tutorials/2012/03/21/deploying-elasticsearch-with-chef-solo.html

echo -e "\nInstalling development dependencies, Ruby and essential tools..." \
"\n===============================================================================\n"
yum install gcc gcc-c++ make automake install ruby-devel libcurl-devel libxml2-devel libxslt-devel vim curl git -y
echo -e "\nInstalling Rubygems..." \
"\n===============================================================================\n"
yum install rubygems -y
gem install json --no-ri --no-rdoc
echo -e "\nInstalling and bootstrapping Chef..." \
"\n===============================================================================\n"
test -d "/opt/chef" || curl -# -L http://www.opscode.com/chef/install.sh | sudo bash -s -- -v 11.6.0
mkdir -p /etc/chef/
mkdir -p /var/chef-solo/site-cookbooks
mkdir -p /var/chef-solo/cookbooks
if test -f /tmp/solo.rb; then mv /tmp/solo.rb /etc/chef/solo.rb; fi
echo -e "\nDownloading cookbooks..." \
"\n===============================================================================\n"
test -d /var/chef-solo/site-cookbooks/monit || curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/915/original/monit.tgz | tar xz -C /var/chef-solo/site-cookbooks/
test -d /var/chef-solo/site-cookbooks/ark || git clone git://github.com/bryanwb/chef-ark.git /var/chef-solo/site-cookbooks/ark
if [ ! -d /var/chef-solo/cookbooks/elasticsearch ]; then
git clone git://github.com/elasticsearch/cookbook-elasticsearch.git /var/chef-solo/cookbooks/elasticsearch
else
cd /var/chef-solo/cookbooks/elasticsearch
git fetch
git reset origin/master --hard
fi
echo -e "\n*******************************************************************************\n" \
"Bootstrap finished" \
"\n*******************************************************************************\n"
source :rubygems
gem 'rake'
gem 'json'
gem 'fog'
gem 'ansi'
{
"run_list": [ "recipe[monit]",
"recipe[elasticsearch]",
"recipe[elasticsearch::plugins]",
"recipe[elasticsearch::ebs]",
"recipe[elasticsearch::data]",
"recipe[elasticsearch::aws]",
"recipe[elasticsearch::nginx]",
"recipe[elasticsearch::proxy]",
"recipe[elasticsearch::monit]" ],
"elasticsearch" : {
"cluster_name" : "elasticsearch_test_with_chef",
"bootstrap" : { "mlockall" : false },
"discovery" : { "type": "ec2" },
"data_path" : "/usr/local/var/data/elasticsearch/disk1",
"data" : {
"devices" : {
"/dev/sda2" : {
"file_system" : "ext3",
"mount_options" : "rw,user",
"mount_path" : "/usr/local/var/data/elasticsearch/disk1",
"format_command" : "mkfs.ext3",
"fs_check_command" : "dumpe2fs",
"ebs" : {
"size" : 25,
"delete_on_termination" : true,
"type" : "io1",
"iops" : 100
}
}
}
},
"cloud" : {
"aws" : {
"access_key" : "<REPLACE>",
"secret_key" : "<REPLACE>",
"region" : "us-east-1"
},
"ec2" : {
"security_group": "elasticsearch-test"
}
},
"plugins" : {
"karmi/elasticsearch-paramedic" : {}
},
"nginx" : {
"users" : [ { "username" : "USERNAME", "password" : "PASSWORD" } ],
"allow_cluster_api" : true
}
},
"monit" : {
"notify_email" : "<REPLACE WITH YOUR E-MAIL>",
"mail_format" : { "from" : "monit@amazonaws.com", "subject" : "[monit] $SERVICE $EVENT on $HOST", "message" : "$SERVICE $ACTION: $DESCRIPTION" }
}
}
# Patch Monit cookbook problems
mkdir -p /etc/monit/conf.d/
rm -f /etc/monit.conf
touch /etc/monit/monitrc
chmod 700 /etc/monit/monitrc
ln -nfs /etc/monit/monitrc /etc/monit.conf
# Patch Nginx cookbook problems
mkdir -p /etc/nginx/sites-available/
useradd -s /bin/sh -u 33 -U -d /var/www -c Webserver www-data
echo -e "\n*******************************************************************************\n" \
"Patching finished" \
"\n*******************************************************************************\n"
require 'rubygems'
require 'json'
require 'fog'
require 'ansi'
module Provision
class Server
attr_reader :name, :options, :node, :ui
def initialize(options = {})
@options = options
@name = @options.delete(:name)
end
def create!
create_node
tag_node
msg "Waiting for SSH...", :yellow
wait_for_sshd and puts
end
def destroy!
servers = connection.servers.select { |s| s.tags["Name"] == name && s.state != 'terminated' }
if servers.empty?
msg "[!] No instance named '#{name}' found!", :red
exit(1)
end
puts "Will terminate #{servers.size} server(s): ",
servers.map { |s| "* #{s.tags["Name"]} (#{s.id})" }.join("\n"),
"Continue? (y/n)"
exit(0) unless STDIN.gets.strip.downcase == 'y'
servers.each do |s|
@node = s
msg "Terminating #{node.tags["Name"]} (#{node.id})...", :yellow
connection.terminate_instances(node.id)
msg "Done.", :green
end
end
def connection
@connection ||= Fog::Compute.new(
:provider => 'AWS',
:aws_access_key_id => options[:aws_access_key_id],
:aws_secret_access_key => options[:aws_secret_access_key],
:region => options[:aws_region]
)
end
def node
@node ||= connection.servers.select do |s|
s.tags["Name"] =~ Regexp.new(name) && s.state != 'terminated'
end.first
end
def create_node
msg "Creating EC2 instance #{name} in #{options[:aws_region]}...", :bold
msg "-"*ANSI::Terminal.terminal_width
@node = connection.servers.create(:image_id => options[:aws_image],
:groups => options[:aws_groups].split(",").map {|x| x.strip},
:flavor_id => options[:aws_flavor],
:key_name => options[:aws_ssh_key_id],
:block_device_mapping => options[:block_device_mapping] || [ { "DeviceName" => "/dev/sde1", "VirtualName" => "ephemeral0" }]
)
msg_pair "Instance ID", node.id
msg_pair "Flavor", node.flavor_id
msg_pair "Image", node.image_id
msg_pair "Region", options[:aws_region]
msg_pair "Availability Zone", node.availability_zone
msg_pair "Security Groups", node.groups.join(", ")
msg_pair "SSH Key", node.key_name
msg "Waiting for instance...", :yellow
@node.wait_for { print "."; ready? }
puts
msg_pair "Public DNS Name", node.dns_name
msg_pair "Public IP Address", node.public_ip_address
msg_pair "Private DNS Name", node.private_dns_name
msg_pair "Private IP Address", node.private_ip_address
end
def tag_node
msg "Tagging instance in EC2...", :yellow
custom_tags = options[:tags].split(",").map {|x| x.strip} rescue []
tags = Hash[*custom_tags]
tags["Name"] = @name
tags.each_pair do |key, value|
connection.tags.create :key => key, :value => value, :resource_id => @node.id
msg_pair key, value
end
end
def wait_for_sshd
hostname = node.dns_name
loop do
begin
print(".")
tcp_socket = TCPSocket.new(hostname, 22)
readable = IO.select([tcp_socket], nil, nil, 5)
if readable
msg "\nSSHd accepting connections on #{hostname}, banner is: #{tcp_socket.gets}", :green
return true
end
rescue SocketError
sleep 2
retry
rescue Errno::ETIMEDOUT
sleep 2
retry
rescue Errno::EPERM
return false
rescue Errno::ECONNREFUSED
sleep 2
retry
rescue Errno::EHOSTUNREACH
sleep 2
retry
ensure
tcp_socket && tcp_socket.close
end
end
end
def ssh(command)
host = node.dns_name
user = options[:ssh_user]
key = options[:ssh_key]
opts = "-o User=#{user} -o IdentityFile=#{key} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
system "ssh -t #{opts} #{host} #{command}"
end
def scp(files, params={})
host = node.dns_name
user = options[:ssh_user]
key = options[:ssh_key]
opts = "-o User=#{user} -o IdentityFile=#{key} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -r"
path = params[:path] || '/tmp'
__command = "scp #{opts} #{files} #{host}:#{path}"
puts __command
system __command
end
def msg message, color=:white
puts message.ansi(color)
end
def msg_pair label, value, color=:cyan
puts (label.to_s.ljust(25) + value.to_s).ansi(color)
end
end
end
desc "Create, bootstrap and configure an instance in EC2"
task :create => :setup do
@server = Provision::Server.new @args
@server.create!
Rake::Task[:upload].execute
Rake::Task[:provision].execute
end
desc "Terminate an EC2 instance"
task :destroy => :setup do
@server = Provision::Server.new @args
@server.destroy!
end
desc "(Re-)provision an instance"
task :provision => :setup do
@server ||= Provision::Server.new @args
@server.ssh "sudo bash /tmp/bootstrap.sh"
@server.ssh "sudo bash /tmp/patches.sh"
@server.ssh "sudo chef-solo -N #{@server.name} -j /tmp/#{@args[:node_json]}"
exit(1) unless $?.success?
puts "_"*ANSI::Terminal.terminal_width
puts "\nOpen " + "http://#{@args[:http_username]}:#{@args[:http_password]}@#{@server.node.dns_name}:8080".ansi(:bold) + " in your browser"
end
task :upload => :setup do
@server ||= Provision::Server.new @args
@server.scp "bootstrap.sh patches.sh #{@args[:node_json]} solo.rb", :path => '/tmp'
exit(1) unless $?.success?
end
task :setup do
ENV['AWS_ACCESS_KEY'] || ( puts "\n[!] Missing AWS_ACCESS_KEY environment variable...".ansi(:red) and exit(1) )
ENV['AWS_SECRET_ACCESS_KEY'] || ( puts "\n[!] Missing AWS_SECRET_ACCESS_KEY environment variable...".ansi(:red) and exit(1) )
node_json = ENV['NODE'] || 'node.json'
json = JSON.parse(File.read( File.expand_path("../#{node_json}", __FILE__) ))
name = json['elasticsearch']['node_name'] rescue nil
aws_access_key = json['elasticsearch']['cloud']['aws']['access_key'] rescue nil
aws_secret_access_key = json['elasticsearch']['cloud']['aws']['secret_key'] rescue nil
aws_region = json['elasticsearch']['cloud']['aws']['region'] rescue nil
aws_group = json['elasticsearch']['cloud']['ec2']['security_group'] rescue nil
http_username = json['elasticsearch']['nginx']['users'][0]['username'] rescue nil
http_password = json['elasticsearch']['nginx']['users'][0]['password'] rescue nil
@args = {}
@args[:name] = ENV['NAME'] || name || 'elasticsearch-test'
@args[:node_json] = node_json
@args[:aws_ssh_key_id] = ENV['AWS_SSH_KEY_ID'] || 'elasticsearch-test'
@args[:aws_access_key_id] = ENV['AWS_ACCESS_KEY_ID'] || aws_access_key
@args[:aws_secret_access_key] = ENV['AWS_SECRET_ACCESS_KEY'] || aws_secret_access_key
@args[:aws_region] = ENV['AWS_REGION'] || aws_region || 'us-east-1'
@args[:aws_groups] = ENV['GROUP'] || aws_group || 'elasticsearch-test'
@args[:aws_flavor] = ENV['FLAVOR'] || 't1.micro'
@args[:aws_image] = ENV['IMAGE'] || 'ami-1624987f'
@args[:ssh_user] = ENV['SSH_USER'] || 'ec2-user'
@args[:ssh_key] = ENV['SSH_KEY'] || File.expand_path('../tmp/elasticsearch-test.pem', __FILE__)
@args[:http_username] = http_username
@args[:http_password] = http_password
end
file_cache_path "/var/chef-solo"
cookbook_path ["/var/chef-solo/site-cookbooks", "/var/chef-solo/cookbooks"]
@tamlyn
Copy link

tamlyn commented Jun 3, 2014

With the latest Amazon Linux AMI (ami-2918e35e) I had to run sudo yum groupinstall "Development Tools" on the server first otherwise Chef would fail on chef_gem[fog].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment