Setting up an Elasticsearch 2.x cluster on EC2

setupES21

Building an ES image on AWS:

Provision an amazon-linux instance through the console with a 8gb root drive and a 80+gb ebs data drive

A) update
sudo yum update

B) mount a data volume 
lsblk # find volume tag assuming it is /dev/xvdb
#check if data
sudo file -s /dev/xvdb #if output is `data`, no filesystem, all good
#make filesystem
sudo mkfs -t ext4 /dev/xvdb
#mount (mkdir the folder first)
sudo mount /dev/xvdb /elasticsearch
# auto-mount on reboot
sudo cp /etc/fstab /etc/fstab.orig #backup fstab
sudo nano /etc/fstab
#append at the end:
/dev/xvdb       /elasticsearch  ext4  defaults,nofail   0   0
#Assuming the volume is mounted after a reboot, this is fine

C) Install Java8 (if not default) (Optional)
sudo yum install  java-1.8.0-openjdk.x86_64
sudo alternatives --config java 
#and then select java8 as the default version

D) Install elasticsearch
# Download and install the public signing key
sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
# Add the elasticsearch repo
sudo nano /etc/yum.repos.d/elasticsearch.repo
# and paste
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
# Install es
sudo yum install elasticsearch
#!!!Important. This creates an elasticsearch user. To access the mounted volume, we need to transfer ownership (or change permissions or other linux voodoo)
sudo chown -R elasticsearch:elasticsearch /elasticsearch


E) A few plugins (https://www.elastic.co/guide/en/elasticsearch/plugins/current/api.html)
# cd to ES_HOME (defaults to /usr/share/elasticsearch on amazon0linux)
#for aws cloud plugin
sudo bin/plugin install cloud-aws
# elasticsearch-sql
sudo bin/plugin install https://github.com/NLPchina/elasticsearch-sql/releases/download/2.1.0/elasticsearch-sql-2.1.0.zip 
# Monitoring (ElasticHQ)
sudo bin/plugin -install royrusso/elasticsearch-HQ
# todo: marvel

F) Configuration
#Edit the elasticsearch.yml config file. default location : /etc/elasticsearch elasticsearch.yml
#Paste >
#Cloud discovery

cloud.aws.access_key: <access_key> #Make sure associated IAM has ec2 permissions
cloud.aws.secret_key: <secret_key>
cloud.aws.region: <region> # ie eu-west-1
cluster.name: <cluster_name> # eg elasticsearch-production
discovery.ec2.groups: <security_group> #sec group of the cluster instances. Makesure TCP 9200-9300 are open to the sec group itself!
discovery.ec2.host_type: "private_ip" #default
discovery.type: "ec2"
discovery.zen.ping.multicast.enabled: false
plugin.mandatory: "cloud-aws"

#Minimum nodes in a quorum / must be set for production systems
discovery.zen.minimum_master_nodes: 2 # nodes/2 + 1. For a 3-node it should be 2, for a 5-node 3 etc

#Node namel
node.name: "Aris"

#Scripting
script.inline: on
script.indexed: on

#Memory ES_HEAP_SIZE must be set to 50% of the total mem
bootstrap.mlockall: true

#Paths
path.data: /elasticsearch/data
path.logs: /elasticsearch/logs

#Network
network.host: "_ec2:publicDns_" #where to bind, def is localhost


It is important at this stage to launch sense, index a few documents and check that everything is working as it should

At this point we can register an AMI!

G) Building a cluster

We just need to launch an isntance using the AMI we just registered and put it in the security group used in discovery.ec2.groups!