ElasticSearch bootstrap script and node configuration for Ubuntu [https://gist.github.com/2050769]

.gitignore

.DS_Store
Gemfile.lock
*.pem
node.json
tmp/*
!tmp/.gitignore

__readme.markdown

Helper files for https://gist.github.com/2050769

This repository contains files tailored for bootstrapping, installing and configuring Elasticsearch with Chef Solo on the Ubuntu operating system, as a supplementary material for the Deploying Elasticsearch with Chef Solo tutorial.

After you have downloaded the files from the gist, copy and edit the node configuration:

cp node_ubuntu.json node.json
$EDITOR node.json

Set up the SSH connection according to your environment, eg.:

HOST=<REPLACE WITH YOUR SERVER IP>
KEY=<REPLACE WITH PATH TO YOUR SSH KEY>
USER=ubuntu
PORT=22
SSH_OPTIONS="-o User=$USER -o IdentityFile=$KEY -o Port=$PORT -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

And test that it's working:

ssh $SSH_OPTIONS $HOST "date"

Now, copy the files to the machine and execute the bootstrap script:

scp $SSH_OPTIONS bootstrap_ubuntu.sh node.json solo.rb $HOST:/tmp

time ssh -t $SSH_OPTIONS $HOST "sudo bash /tmp/bootstrap_ubuntu.sh"

Execute the Chef Solo run:

time ssh -t $SSH_OPTIONS $HOST "sudo su - root -c 'chef-solo --node-name elasticsearch-test-ubuntu -j /tmp/node.json'"

Check that the installation and configuration succeeded (you may need to wait couple of minutes until Monit starts):

ssh -t $SSH_OPTIONS $HOST "sudo service elasticsearch status -v"

ssh -t $SSH_OPTIONS $HOST "sudo monit reload && sudo monit status -v"

curl http://USERNAME:PASSWORD@$HOST:8080

Now, you can eg. test that Monit is, in fact, working. Kill the Elasticsearch process:

ssh -t $SSH_OPTIONS $HOST "cat '/usr/local/var/run/elasticsearch/elasticsearch_test_ubuntu.pid' | xargs -0 sudo kill -9"

Wait couple of minutes and check the status:

ssh -t $SSH_OPTIONS $HOST "sudo monit reload && sudo monit status -v"

See the original gist for more information.

---

http://www.elasticsearch.org/tutorials/2012/03/21/deploying-elasticsearch-with-chef-solo.html

bootstrap_ubuntu.sh

echo -e "\nInstalling development dependencies and essential tools..." \
        "\n===============================================================================\n"
# Update packages
apt-get update --yes --fix-missing
# Install build tools
apt-get install build-essential --yes --fix-missing
# Install utilities
apt-get install vim screen curl git-core --yes --fix-missing
# Make sure we are running a Bash shell
ln -sf /bin/bash /bin/sh

echo -e "\nInstalling Ruby and Rubygems..." \
        "\n===============================================================================\n"
# Install Ruby dependencies
apt-get install bison zlib1g-dev libopenssl-ruby1.9.1 libssl-dev libreadline5-dev libncurses5-dev libyaml-0-2 libxslt-dev libxml2-dev file --yes --fix-missing
# Install Ruby
apt-get install ruby1.9.1 ruby1.9.1-dev --yes --fix-missing
# Install Rubygems
apt-get install rubygems1.9.1 --yes --fix-missing
# "Fix Debian"
ln -nfs /usr/bin/ruby1.9.1 /usr/local/bin/ruby
echo "PATH=$PATH:/var/lib/gems/1.9.1/bin/" > /etc/profile.d/rubygems.sh
source /etc/profile.d/rubygems.sh
# Install the JSON gem
gem install json --no-ri --no-rdoc

echo -e "\nInstalling and bootstrapping Chef..." \
        "\n===============================================================================\n"
test -d "/opt/chef" || curl -# -L http://www.opscode.com/chef/install.sh | sudo bash -s --

mkdir -p /etc/chef/
mkdir -p /var/chef-solo/site-cookbooks
mkdir -p /var/chef-solo/cookbooks

if test -f /tmp/solo.rb;   then mv /tmp/solo.rb /etc/chef/solo.rb;     fi
if test -d /tmp/data_bags; then mv /tmp/data_bags /etc/chef/data_bags; fi

echo -e "\nDownloading cookbooks..." \
        "\n===============================================================================\n"
test -d /var/chef-solo/site-cookbooks/java || curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/2811/original/java.tgz | tar xz -C /var/chef-solo/site-cookbooks/

test -d /var/chef-solo/site-cookbooks/monit || curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/915/original/monit.tgz | tar xz -C /var/chef-solo/site-cookbooks/

test  -d /var/chef-solo/site-cookbooks/nginx || curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1413/original/nginx.tgz | tar xz -C /var/chef-solo/site-cookbooks

test  -d /var/chef-solo/site-cookbooks/ark ||  curl -# -L -k http://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/1631/original/ark.tgz | tar xz -C /var/chef-solo/site-cookbooks

if [ ! -d /var/chef-solo/cookbooks/elasticsearch ]; then
  git clone git://github.com/elasticsearch/cookbook-elasticsearch.git /var/chef-solo/cookbooks/elasticsearch
else
  cd /var/chef-solo/cookbooks/elasticsearch
  git fetch
  git reset origin/master --hard
fi

echo -e "\n*******************************************************************************\n" \
        "Bootstrap finished" \
        "\n*******************************************************************************\n"

node_ubuntu.json

{
  "run_list": [ "recipe[monit]",
                "recipe[java]",
                "recipe[nginx]",
                "recipe[elasticsearch]",
                "recipe[elasticsearch::proxy]",
                "recipe[elasticsearch::monit]" ],

  "elasticsearch" : {
    "cluster_name" : "elasticsearch_test_with_chef",
    "mlockall"     : false,

    "nginx" : {
      "user"  : "www-data",
      "users" : [ { "username" : "USERNAME", "password" : "PASSWORD" } ]
    }
  },

  "monit" : {
    "notify_email" : "<REPLACE WITH YOUR E-MAIL>",
    "mail_format"  : { "from" : "monit@example.com", "subject" : "[monit] $SERVICE $EVENT on $HOST", "message" : "$SERVICE $ACTION: $DESCRIPTION" }
  }

}

solo.rb

file_cache_path "/var/chef-solo"
cookbook_path   ["/var/chef-solo/site-cookbooks", "/var/chef-solo/cookbooks"]

It would be best to use https://www.opscode.com/chef/install.sh on line 29 above.

I hit a bunch of errors running this gist. First, a whole bunch of dependencies seem not to be properly included in the elasticsearch recipe. I needed to add them all manually in the bootstrap.sh.

Second, monit just doesn't work. I had to disable the recipe to get the whole thing to install.

Adding these lines to the bootstrap installs the missing dependencies:

if [ ! -d /var/chef-solo/cookbooks/windows ]; then
git clone git://github.com/opscode-cookbooks/windows.git /var/chef-solo/cookbooks/windows
else
cd /var/chef-solo/cookbooks/windows
git fetch
git reset origin/master --hard
fi

if [ ! -d /var/chef-solo/cookbooks/chef_handler ]; then
git clone git://github.com/opscode-cookbooks/chef_handler.git /var/chef-solo/cookbooks/chef_handler
else
cd /var/chef-solo/cookbooks/chef_handler
git fetch
git reset origin/master --hard
fi

if [ ! -d /var/chef-solo/cookbooks/bluepill ]; then
git clone git://github.com/opscode-cookbooks/bluepill.git /var/chef-solo/cookbooks/bluepill
else
cd /var/chef-solo/cookbooks/bluepill
git fetch
git reset origin/master --hard
fi

if [ ! -d /var/chef-solo/cookbooks/rsyslog ]; then
git clone git://github.com/opscode-cookbooks/rsyslog.git /var/chef-solo/cookbooks/rsyslog
else
cd /var/chef-solo/cookbooks/rsyslog
git fetch
git reset origin/master --hard
fi

if [ ! -d /var/chef-solo/cookbooks/build-essential ]; then
git clone git://github.com/opscode-cookbooks/build-essential.git /var/chef-solo/cookbooks/build-essential
else
cd /var/chef-solo/cookbooks/build-essential
git fetch
git reset origin/master --hard
fi

if [ ! -d /var/chef-solo/cookbooks/runit ]; then
git clone git://github.com/opscode-cookbooks/runit.git /var/chef-solo/cookbooks/runit
else
cd /var/chef-solo/cookbooks/runit
git fetch
git reset origin/master --hard
fi

if [ ! -d /var/chef-solo/cookbooks/yum ]; then
git clone git://github.com/opscode-cookbooks/yum.git /var/chef-solo/cookbooks/yum
else
cd /var/chef-solo/cookbooks/yum
git fetch
git reset origin/master --hard
fi

To avoid having to specify all the dependencies, put in the chef version to be the same as in the original bootstrap.sh:

test -d "/opt/chef" || curl -# -L http://www.opscode.com/chef/install.sh | sudo bash -s -- -v 10.18.2

I haven't gotten monit to work yet, but I don't want to spend much time on that yet.

If you want it to autoscale, it's also missing the complete configuration from the other gist. My guess is you can copy node-example.json from there and try that. I'll get back with more details after I get autodiscovery running.

Update: I switched to Amazon Linux, it's a lot easier to just use the Rakefile from the original gist. After a couple of fixes it will just work.