Bootstrap, install and configure ElasticSearch with Chef Solo

.gitignore

.DS_Store
node.json
tmp/

__readme.markdown

Bootstrap, install and configure ElasticSearch with Chef Solo

Download or clone the files in this gist:

curl -# -L -k https://gist.github.com/gists/2050769/download | tar xz --strip 1 -C .

First, in the downloaded node.json file, replace the bucket value with a unique name, and the access_key and secret_key values with proper credentials.

Second, create a dedicated security group in the AWS console for ElasticSearch nodes. We will be using group named elasticsearch-test.

Make sure the security groups allows connections on following ports:

• Port 22 for SSH is open for external access (the default 0.0.0.0/0)
• Port 8080 for the Nginx proxy is open for external access (the default 0.0.0.0/0)
• Port 9300 for in-cluster communication is open to the same security group (use the Group ID for this group, available on the "Details" tab, such as sg-1a23bcd)

Third, launch a new instance in the AWS console:

• Use a meaningful name for the instance. We will use elasticsearch-test-chef-1.
• Create a new "Key Pair" for the instance, and download it. We will be using a key named elasticsearch-test.
• Use the Amazon Linux AMI (ami-1b814f72). Amazon Linux comes with Ruby and Java pre-installed.
• Use the m1.large instance type. You may use the small or even micro instance type, but the process will take very long, due to AWS constraints (could be hours instead of minutes).
• Use the security group created in the first step (elasticsearch-test).

Copy the SSH key downloaded from AWS console to the tmp/ directory of this project and change its permissions:

cp ~/Downloads/elasticsearch-test.pem ./tmp
chmod 600 ./tmp/elasticsearch-test.pem

Once the instance is ready, copy its "Public DNS" in the AWS console (eg. ec2-123-40-123-50.compute-1.amazonaws.com ).

We can begin the bootstrap and install process now.

Let's setup the connection details, first:

HOST=<REPLACE WITH YOUR PUBLIC DNS>
SSH_OPTIONS="-o User=ec2-user -o IdentityFile=./tmp/elasticsearch-test.pem -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

Let's copy the files to the machine:

scp $SSH_OPTIONS ./bootstrap.sh ./patches.sh ./node.json ./data_bags ./solo.rb $HOST:/tmp

Let's bootstrap the machine (ie. install neccessary packages, download cookbooks, etc):

time ssh -t $SSH_OPTIONS $HOST "sudo bash /tmp/bootstrap.sh"
time ssh -t $SSH_OPTIONS $HOST "sudo bash /tmp/patches.sh"

The following command will run the chef-solo command to provision the system:

time ssh -t $SSH_OPTIONS $HOST "sudo chef-solo -N elasticsearch-test-1 -j /tmp/node.json"

After the Chef run succeeded, you can check whether ElasticSearch is running on the machine (leave couple of seconds for ElasticSearch to have a chance to start...):

ssh -t $SSH_OPTIONS $HOST "curl localhost:9200/_cluster/health?pretty"

You can also connect to the Nginx-based proxy:

curl http://USERNAME:PASSWORD@$HOST:8080

And use it for indexing some data:

curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/document/1" -d '{"title" : "Test 1"}'
curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/document/1" -d '{"title" : "Test 2"}'
curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/document/1" -d '{"title" : "Test 3"}'
curl -X POST "http://USERNAME:PASSWORD@$HOST:8080/test_chef_cookbook/_refresh"

Or performing searches:

curl "http://USERNAME:PASSWORD@$HOST:8080/_search?pretty&q=*"

You can also use the provided service to check ElasticSearch status:

ssh -t $SSH_OPTIONS $HOST "sudo service elasticsearch status -v"

Of course, you can check the ElasticSearch status with Monit:

ssh -t $SSH_OPTIONS $HOST "sudo monit reload && sudo monit status -v"

(If the Monit daemon is not running, start it with sudo service monit start first. Notice the daemon has a startup delay of 2 minutes by default.)

bootstrap.sh

echo -e "\nInstalling development dependencies and essential tools..." \
        "\n===============================================================================\n"
yum install gcc gcc-c++ make automake install ruby-devel libcurl-devel libxml2-devel libxslt-devel vim curl git -y

echo -e "\nInstalling Rubygems..." \
        "\n===============================================================================\n"
yum install rubygems -y

echo -e "\nInstalling and bootstrapping Chef..." \
        "\n===============================================================================\n"
if ! which chef-solo 2>&1 > /dev/null; then gem install chef --version 0.10.8 --no-rdoc --no-ri; fi

mkdir -p /etc/chef/
mkdir -p /var/chef-solo/site-cookbooks
mkdir -p /var/chef-solo/cookbooks

if test -f /tmp/solo.rb; then mv /tmp/solo.rb /etc/chef/solo.rb; fi

echo -e "\nDownloading cookbooks..." \
        "\n===============================================================================\n"
if ! test -d /var/chef-solo/site-cookbooks/monit; then
curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/915/original/monit.tgz | tar xz -C /var/chef-solo/site-cookbooks/
fi

if ! test  -d /var/chef-solo/site-cookbooks/nginx; then
curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1413/original/nginx.tgz | tar xz -C /var/chef-solo/site-cookbooks
fi

if ! test -d /var/chef-solo/cookbooks/elasticsearch; then
git clone git://github.com/karmi/cookbook-elasticsearch.git /var/chef-solo/cookbooks/elasticsearch
else
cd /var/chef-solo/cookbooks/elasticsearch
git fetch
git reset origin/master --hard
fi

echo -e "\n*******************************************************************************\n" \
        "Bootstrap finished" \
        "\n*******************************************************************************\n"

node-example.json

{
  "run_list": [ "recipe[monit]",
                "recipe[nginx]",
                "recipe[elasticsearch]",
                "recipe[elasticsearch::plugin_aws]",
                "recipe[elasticsearch::proxy_nginx]" ],

  "elasticsearch" : {
    "cluster_name" : "elasticsearch_test_with_chef",

    "discovery" : { "type": "ec2" },

    "gateway" : {
      "type"    : "s3",
      "s3"      : { "bucket": "<REPLACE WITH A UNIQUE NAME SUCH AS yourname-elasticsearch-test-with-chef>" }
    },

    "cloud"   : {
      "aws"     : { "access_key": "<REPLACE>", "secret_key": "<REPLACE>" },
      "ec2"     : { "security_group": "elasticsearch-test" }
    },

    "nginx" : {
      "users" : [ { "username" : "USERNAME", "password" : "PASSWORD" } ]
    }
  },

  "monit" : {
    "notify_email" : "<REPLACE WITH YOUR E-MAIL>",
    "mail_format"  : { "from" : "monit@amazonaws.com", "subject" : "[monit] $SERVICE $EVENT on $HOST", "message" : "$SERVICE $ACTION: $DESCRIPTION" }
  }

}

patches.sh

# Patch Monit cookbook problems
mkdir -p /etc/monit/conf.d/
rm -f /etc/monit.conf
touch /etc/monit/monitrc
chmod 700 /etc/monit/monitrc
ln -nfs /etc/monit/monitrc /etc/monit.conf

# Patch Nginx cookbook problems
mkdir -p /etc/nginx/sites-available/
useradd -s /bin/sh -u 33 -U -d /var/www -c Webserver www-data

solo.rb

file_cache_path "/var/chef-solo"
cookbook_path   ["/var/chef-solo/site-cookbooks", "/var/chef-solo/cookbooks"]