ElasticSearch bootstrap script and node configuration for Ubuntu [https://gist.github.com/2050769]

__readme.markdown

Installing ElasticSearch with Monit & nginx

This repository contains files tailored for bootstrapping, installing and configuring ElasticSearch with Chef Solo on the Ubuntu operating system, inspired by this article.

Launch and ssh into Ubuntu 12.04 instance (if on ec2, use ami from alestic.com and with a security group as detailed here)

Now, copy the files to the machine and execute the bootstrap script:

mkdir /tmp
cd /tmp
curl -# -L -k https://gist.github.com/gists/4200167/download | tar xz --strip 1 -C .

sudo -H bash /tmp/bootstrap_ubuntu.sh

After you have downloaded the files, edit the node configuration:

$EDITOR node_ubuntu.json

Execute the Chef Solo run:

sudo su - root -c 'chef-solo --node-name elasticsearch-test-ubuntu -j /tmp/node_ubuntu.json'

Check that the installation and configuration succeeded (you may need to wait couple of minutes until Monit starts):

sudo service elasticsearch status -v

sudo monit reload && sudo monit status -v

Now, you can eg. test that Monit is, in fact, working. Kill the ElasticSearch process:

cat '/usr/local/var/run/elasticsearch/elasticsearch_test_ubuntu.pid' | xargs -0 sudo kill -9

Wait couple of minutes and check the status:

sudo monit reload && sudo monit status -v

Logout and check if you're able to connect.

curl http://USERNAME:PASSWORD@$HOST:8080

See the original gist for more information.

bootstrap_ubuntu.sh

echo -e "\nInstalling development dependencies and essential tools..." \
        "\n===============================================================================\n"
# Update packages
apt-get update --yes --fix-missing
apt-get upgrade -y
# Install build tools
apt-get install build-essential --yes --fix-missing
# Install utilities
apt-get install vim screen curl git-core --yes --fix-missing
# Make sure we are running a Bash shell
ln -sf /bin/bash /bin/sh

echo -e "\nInstalling Ruby and Rubygems..." \
        "\n===============================================================================\n"
# Install Ruby dependencies
apt-get install bison zlib1g-dev libopenssl-ruby1.9.1 libssl-dev libreadline5-dev libncurses5-dev libyaml-0-2 file --yes --fix-missing
# Install Ruby
apt-get install ruby1.9.1 ruby1.9.1-dev --yes --fix-missing
# Install Rubygems
apt-get install rubygems1.9.1 --yes --fix-missing
# "Fix Debian"
ln -nfs /usr/bin/ruby1.9.1 /usr/local/bin/ruby
echo "PATH=$PATH:/var/lib/gems/1.9.1/bin/" > /etc/profile.d/rubygems.sh
source /etc/profile.d/rubygems.sh
# Install the JSON gem
gem install json --no-ri --no-rdoc

echo -e "\nInstalling and bootstrapping Chef..." \
        "\n===============================================================================\n"
test -d "/opt/chef" || curl -# -L http://www.opscode.com/chef/install.sh | sudo bash -s -- -v 10.14.4

mkdir -p /etc/chef/
mkdir -p /var/chef-solo/site-cookbooks
mkdir -p /var/chef-solo/cookbooks

if test -f /tmp/solo.rb;   then mv /tmp/solo.rb /etc/chef/solo.rb;     fi
if test -d /tmp/data_bags; then mv /tmp/data_bags /etc/chef/data_bags; fi

echo -e "\nDownloading cookbooks..." \
        "\n===============================================================================\n"
if ! test -d /var/chef-solo/site-cookbooks/monit; then
curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/915/original/monit.tgz | tar xz -C /var/chef-solo/site-cookbooks/
fi

curl -# -L -k https://github.com/opscode-cookbooks/ark/archive/master.tar.gz  | tar xz -C /var/chef-solo/site-cookbooks/
mv /var/chef-solo/site-cookbooks/ark-master/ /var/chef-solo/site-cookbooks/ark

curl -# -L -k https://github.com/opscode-cookbooks/java/archive/master.tar.gz  | tar xz -C /var/chef-solo/site-cookbooks/
mv /var/chef-solo/site-cookbooks/java-master/ /var/chef-solo/site-cookbooks/java

curl -# -L -k https://github.com/opscode-cookbooks/apt/archive/master.tar.gz  | tar xz -C /var/chef-solo/site-cookbooks/
mv /var/chef-solo/site-cookbooks/apt-master/ /var/chef-solo/site-cookbooks/apt

if ! test  -d /var/chef-solo/site-cookbooks/nginx; then
curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1413/original/nginx.tgz | tar xz -C /var/chef-solo/site-cookbooks
fi

if ! test -d /var/chef-solo/cookbooks/elasticsearch; then
git clone git://github.com/karmi/cookbook-elasticsearch.git /var/chef-solo/cookbooks/elasticsearch
else
cd /var/chef-solo/cookbooks/elasticsearch
git fetch
git reset origin/master --hard
fi

echo -e "\n*******************************************************************************\n" \
        "Bootstrap finished" \
        "\n*******************************************************************************\n"



#########
# patch #
#########

# Patch Monit cookbook problems
mkdir -p /etc/monit/conf.d/
rm -f /etc/monit.conf
touch /etc/monit/monitrc
chmod 700 /etc/monit/monitrc
ln -nfs /etc/monit/monitrc /etc/monit.conf

# Patch Nginx cookbook problems
mkdir -p /etc/nginx/sites-available/
useradd -s /bin/sh -u 33 -U -d /var/www -c Webserver www-data
useradd -s /bin/false nginx

wget -O /etc/chef/solo.rb https://raw.github.com/gist/2050769/ee2c40b7713982fa9cc4adc6603cdada1ec45f3d/solo.rb


echo -e “\n*******************************************************************************\n” \
        “Patching finished” \
        “\n*******************************************************************************\n”

node_ubuntu.json

{
  "run_list": [ "recipe[apt]",
                "recipe[monit]",
                "recipe[java]",
                "recipe[nginx]",
                "recipe[elasticsearch]",
                "recipe[elasticsearch::proxy]" ],

  "elasticsearch" : {
    "cluster_name" : "elasticsearch_test_with_chef",
    "mlockall"     : false,


    “discovery” : { “type”: “ec2” },

    “gateway” : {
      “type”    : “s3”,
      “s3”      : { “bucket”: “<REPLACE WITH A UNIQUE NAME SUCH AS yourname-elasticsearch-test-with-chef>” }
    },

    “cloud”   : {
      “aws”     : { “access_key”: “<REPLACE>”, “secret_key”: “<REPLACE>” },
      “ec2”     : { “security_group”: “elasticsearch-test” }
    },


    "nginx" : {
      "user"  : "www-data",
      "users" : [ { "username" : "USERNAME", "password" : "PASSWORD" } ]
    }
  },

  "monit" : {
    "notify_email" : "<REPLACE WITH YOUR E-MAIL>",
    "mail_format"  : { "from" : "monit@example.com", "subject" : "[monit] $SERVICE $EVENT on $HOST", "message" : "$SERVICE $ACTION: $DESCRIPTION" }
  }

}

sg.markdown

Port 22 for SSH is open for external access (the default 0.0.0.0/0)

Port 8080 for the Nginx proxy is open for external access (the default 0.0.0.0/0)

Port 9300 for in-cluster communication is open for access only to servers running in the same security group (use the “Group ID” for this group, available on the “Details” tab, such as sg-1a23bcd)

Also, add a firewall rule to allow a custom ICMP rule “destination unreachable” (under Custom ICMP rules) “fragmentation needed” (see dropdown) packets to pass. For more details look here

---

Tip: Use 10.0.0.0/8 to restrict access to only ec2 instances.

solo.rb

file_cache_path "/var/chef-solo"
cookbook_path   ["/var/chef-solo/site-cookbooks", "/var/chef-solo/cookbooks"]