##Deploying an Elastic Cluster on Azure This document assumes you already have an azure subscription. We'll use a pre-configured azure resource manager template built by the Elastic team to simplify the deployment of an Elastic cluster on Azure. By using this template, you can specify the number of data / master nodes, install commercial cluster oversight and monitoring products such as Marvel for cluster / index monitoring, Kibana for data analytics and insight, Shield for authentication and Sense for an intuitive light weight data querying tool.
We'll be using a Bring-Your-Own-License (BYOL) solution template. Elasticsearch and Kibana are both open source and free of use. Marvel basic monitoring is covered under the free basic subscription, which you can sign up for here. ###Lets get started
-
Use the Elastic and Kibana ARM template here to get started. This template will also create and automount a 4TB data volume to all your nodes via Azure File Storage. You can configure this once your cluster is fully allocated.
-
Here's a screenshot of my basic settings, which I chose my authentication type as password.
- I'm using the latest Elastic 2.3.4 build for my cluster. The template will create all the necessary virtual network group resources, including all the proper security group / elastic port configuration settings.
- Nodes Configration - How many data and client nodes you allocate fully depends on the anticipated data volume and index rate. You should strive to scale horizontally by keeping your client and data nodes on seperate VM's, and adding more nodes to the cluster as you scale out. Elasticsearch shines with allowing you to easily scale out your indexing and searching across a full cluster of machines. For comparison purposes, I created a 6 node cluster(2 master + 2 client + 2 data) for the Cities Unlocked project which managed ~20 million spatial documents from openstreetmap. Here's some more guidance which you may find helpful to provide more insight into what cluster topology works best for you.
- External Access - This template allows you to install Kibana as part of the Elastic bundle, which is a brilliant tool that provides insight and gives a shape to your data. You will need Kibana installed in order to use Marvel to monitor the performance and health of your cluster. Kibana will act as your jumpbox, therefore you can opt out of adding an explicit jumpbox to your cluster. Make sure to select the load balancer type as External.
Confirm the resource summary to commence the Azure resource group allocation, which can take up to 15 minutes.
###Accessing your cluster Once all resources have been allocated, you can access your cluster via Kibana through the public IP DNS address resource in your Azure resource group, as highlighted below(i.e. mykibanaDNS.region.cloudapp.azure.com:5601). Look for the resource with the suffix kibana-ip. The elastic web tools are exposed through port 5601. Use the es_kibana as the username and the password you provided by the ARM template setup.
i.e.
You can use elastic's REST API to fully manage your indexes. Your cluster API endpoint can be found by inspecting the external IP address resource in Azure that's suffixed with external-lb-ip, as highlighted below.
The elastic REST API is exposed via port 9201 :9201. Your cluster is configured with the Shield plugin by default, and setup with basic authentication. You can use Postman as your API interface, and authenticate as es_admin with the password you provided within the ARM template setup.
To verify that you can access your node settings through the elastic REST API try the following GET endpoint request :9201/_nodes/process?pretty.
###Updating your license Your cluster is setup with a 30-day trial license. You can view the license details via their GET API call :9200/_license. Once you've received an email from the elastic team with the key and JSON file of your new license, send a PUT request to :9200/_license?acknowledge=true with the license JSON body as the request body.
Is there up-to-date template for installing es on Azure?