ElasticSearch Cluster Setup Guide

elasticsearch-cluster.md

ElasticSearch Cluster Setup Guide

This document will guide you to provision and installing ElasticSearch in Cluster mode with:

• Master Node
• Ingest Node
• Data Node

When this document writen i'am using Amazon Web Service (AWS) as Cloud Provider with Elastic Cloud Compute (EC2) Instance. I provision EC2 Instance with Operating System Ubuntu 16.04 AMI Code ami-0ee0b284267ea6cde and following type:

• Master & Ingest Node: 3 x t2.large (2 Core, 8 GiB)
• Data Node: 3 x t2.xlarge (4 Core, 16 GiB)

Additional Node Required:

• Bastion Node (for SSH): 1 x t2.micro (1 core, 1GiB)

I also seperate the Virtual Private Cloud (VPC) with Private and Public Subnet where the subnet configuration is configured like following:

• 172.10.0.224/27 (Private Subnet for Availbility Zone A)
• 172.10.0.192/27 (Private Subnet for Availbility Zone B)
• 172.10.0.160/27 (Private Subnet for Availbility Zone C)
• 172.10.0.0/27 (Public Subnet for LB Availibility Zone A)
• 172.10.0.32/27 (Public Subnet for LB Availibility Zone B)
• 172.10.0.64/27 (Public Subnet for LB Availibility Zone C)

And the Security Group like following:

• ElasticSearch-Node
  • Inbound
    • TCP 9200 from ElasticSearch-Node Security Group
    • TCP 9300 from ElasticSearch-Node Security Group
    • TCP 8002 from ElasticSearch-Node Security Group
    • TCP 8002 from ElasticSearch-Node-LB-Internal Security Group
    • TCP 8002 from ElasticSearch-Node-LB-Public Security Group
  • Outbound
    • All TCP/UDP to 0.0.0.0/0
• ElasticSearch-Node-SSH
  • Inbound
    • TCP 22 from ElasticSearch-Node-Bastion Security Group
  • Outbound
    • All TCP/UDP to 0.0.0.0/0
• ElasticSearch-Node-Bastion
  • Inbound
    • TCP 22 from Your Public IP
  • Outbound
    • All TCP/UDP to 0.0.0.0/0
• ElasticSearch-Node-LB-Internal
  • Inbound
    • TCP 80 from Default Security Group
  • Outbound
    • All TCP/UDP to 0.0.0.0/0
• ElasticSearch-Node-LB-Public
  • Inbound
    • TCP 443 from 0.0.0.0/0
  • Outbound
    • All TCP/UDP to 0.0.0.0/0

For the Elastic Block Storage (EBS) every EC2 Instance has 2 disk, the first one is for Root Filesystem and the second one for ZFS Datastore. I use ZFS because i will use ZFS Snapshot for Data Backup in Filesystem Level, you can add AWS EBS Snapshot for Disk Backup.

If any other files that needs to be rewrited it will be available at https://gist.github.com/dimaskiddo/e2135ced6f3e46d6ce11cb5b8fce75e5. Now, we can start the setup with the following step-by-step :

Upgrade Operating System

sudo apt-get -y update
sudo apt-get -y dist-upgrade
sudo apt-get -y install ca-certificates apt-transport-https software-properties-common
sudo apt-get -y install curl wget rsync
sudo apt-get -y install net-tools dnsutils
sudo apt-get -y install htop nano
sudo apt-get -y autoremove
sudo apt-get -y clean

timedatectl set-timezone Asia/Jakarta

Install ZFS

sudo apt-get -y update
sudo apt-get -y install zfsutils-linux
sudo apt-get -y autoremove
sudo apt-get -y clean

sudo fdisk /dev/xvdb
sudo zpool create -O compression=lz4 -o ashift=12 data /dev/xvdb1

sudo zfs create -s -o volblocksize=64k -V <CHANGE_THIS_TO_ZFS_DISK_SIZE> data/elasticsearch
sudo mkfs -t xfs -f /dev/zvol/data/elasticsearch

Install ElasticSearch

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get -y update
sudo apt-get -y install elasticsearch
sudo apt-get -y autoremove
sudo apt-get -y clean

sudo systemctl stop elasticsearch
sudo systemctl disable elasticsearch

sudo nano /etc/fstab
sudo mount /dev/zvol/data/elasticsearch /var/lib/elasticsearch/

sudo mount -a
sudo lsblk

sudo mkdir -p /var/lib/elasticsearch/data
sudo mkdir -p /var/lib/elasticsearch/logs

sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/
sudo chmod -R g+rs /var/lib/elasticsearch/
sudo chmod -R o-rwx /var/lib/elasticsearch/

sudo rm -rf /var/log/elasticsearch
sudo ln -sf /var/lib/elasticsearch/logs /var/log/elasticsearch

sudo mv /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.orig~
sudo nano /etc/elasticsearch/elasticsearch.yml
sudo chmod 660 /etc/elasticsearch/elasticsearch.yml

sudo mkdir -p /etc/systemd/system/elasticsearch.service.d
sudo nano /etc/systemd/system/elasticsearch.service.d/override.conf
sudo systemctl daemon-reload

sudo nano /etc/default/elasticsearch

sudo systemctl restart elasticsearch
sudo systemctl enable elasticsearch

Install Nginx

Nginx installation only required for Ingest Node only it's required as Basic Auth Authentication to replace X-Pack Security Basic Auth module

sudo add-apt-repository ppa:ondrej/nginx

sudo apt-get -y update
sudo apt-get -y install nginx apache2-utils
sudo apt-get -y autoremove
sudo apt-get -y clean

sudo systemctl stop nginx
sudo systemctl disable nginx

sudo unlink /etc/nginx/sites-enabled/default

sudo nano /etc/nginx/nginx.conf
sudo nano /etc/nginx/sites-available/elasticsearch.conf

sudo ln -sf /etc/nginx/sites-available/elasticsearch.conf /etc/nginx/sites-enabled/

sudo htpasswd -c /etc/nginx/elasticsearch.htpasswd admin
sudo nginx -t

sudo systemctl restart nginx
sudo systemctl enable nginx

elasticsearch.conf

server {
  server_name _;

  listen                8002 default_server;
  keepalive_timeout     65;
  types_hash_max_size   2048;
  client_max_body_size  25m;
  charset               utf-8;

  location / {
    proxy_pass http://localhost:9200;
    include proxy_params;

    auth_basic "Authorization Required";
    auth_basic_user_file /etc/nginx/elasticsearch.htpasswd;

    proxy_set_header Authorization "";
  }
}

elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: <CHANGE_THIS_TO_ELASTIC_CLUSTER_NAME>
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: <CHANGE_THIS_TO_ELASTIC_NODE_NAME>-<CHANGE_THIS_TO_ELASTIC_NODE_NUMBER>
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch/data
#
# Path to log files:
#
path.logs: /var/lib/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: <CHANGE_THIS_TO_ELASTIC_NODE_IP>,_local_
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["<CHANGE_THIS_TO_ELASTIC_MASTER_IPS>"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["<CHANGE_THIS_TO_ELASTIC_MASTER_NAME>"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
gateway.recover_after_nodes: 2
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#action.auto_create_index: "*"
#
transport.tcp.port: 9300
#
http.cors.enabled: true
http.cors.allow-origin: "*"
#
xpack.license.self_generated.type: basic
#
discovery.type: zen
discovery.zen.minimum_master_nodes: 2
#
node.master: <CHANGE_THIS_TO_TRUE_IF_ELASTIC_NODE_TYPE_IS_MASTER_OTHERWISE_FALSE>
node.ingest: <CHANGE_THIS_TO_TRUE_IF_ELASTIC_NODE_TYPE_IS_INGEST_OTHERWISE_FALSE>
node.data: <CHANGE_THIS_TO_TRUE_IF_ELASTIC_NODE_TYPE_IS_DATA_OTHERWISE_FALSE>

override.conf

[Service]
LimitMEMLOCK=infinity
LimitNPROC=infinity
LimitNOFILE=infinity
LimitCORE=0