This guide has been revised 06-03-2015. Start with a clean ubuntu 14.04LTS-x64 machine and get it updated
sudo su
apt-get update && apt-get dist-upgrade
reboot
start with on: http://azure.microsoft.com/en-us/documentation/articles/virtual-machines-linux-how-to-attach-disk/ mount the disc as '/elasticsearch-data
mkdir /elasticsearch-data/data
mkdir /elasticsearch-data/logs
#set the owner of the dirs to the elasticsearch service
chown elasticsearch.elasticsearch /elasticsearch-data/data
chown elasticsearch.elasticsearch /elasticsearch-data/logs
#keep away nosey people
chmod 700 /elasticsearch-data/data
chmod 700 /elasticsearch-data/logs
apt-get install openjdk-7-jre-headless -y
Now we can configure apt-get to listen to package update from elasticsearch. Add it to your apt-get command. Following steps are extracted from http://www.elasticsearch.org/blog/apt-and-yum-repositories/ NOTE: 1.2 was latest stable releast at this time (15-07-2014). Update the package source url as required
#get the GPG key for elasticsearch apt-get channel
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
#add package source to your sources.list
nano /etc/apt/sources.list
#nano opens the text editor
#add following lines to sources.list:
deb http://packages.elasticsearch.org/elasticsearch/1.4/debian stable main
#save file / exit nano
apt-get update && apt-get install elasticsearch
Elasticsearch is now up and running. Important paths to remember:
- Configs are stored in
/etc/elasticsearch/ - Elasticsearch resides in
/usr/share/elasticsearch/
More information: http://code972.com/blog/2014/07/74-the-definitive-guide-for-elasticsearch-on-windows-azure
start / stop the elasticsearch service:
/etc/init.d/elasticsearch restart
restart|stop|start
vi /etc/elasticsearch/elasticsearch.yml
cluster.name: my-production-cluster
node.name: "my-node-name"
#determine if you need script acces, it is a possible security risk!
script.disable_dynamic: false
#if you run 2 nodes set it to 2. This will make ES auto start the restore/balancing if 2 nodes are found
gateway.expected_nodes: 2
#if you use unicast, set the ip's of the other nodes in this array
discovery.zen.ping.unicast.hosts: ["10.0.0.1"]
#set your data + logs on an other partition
path.data: /elasticsearch-data/data
path.logs: /elasticsearch-data/logs
vi /etc/init.d/elasticsearch
Add / update the following vars:
ES_MIN_MEM=256m
ES_MAX_MEM=1g
ElasticSearch uses quite a few of file descriptors, both for Lucene indexes and Netty. Raise the number of available file descriptors to the user running ElasticSearch. [http://www.elasticsearch.org/tutorials/too-many-open-files/]
vi /etc/security/limits.conf
# append the following to the file:
elasticsearch soft nofile 32000
elasticsearch hard nofile 32000
update-rc.d elasticsearch defaults 95 10
cd /usr/share/elasticsearch
bin/plugin --install mobz/elasticsearch-head
Last but not least; configure your VM to pass-through connections for your external dns name port 9200 (or whatever you like) to the internal VM 9200 port). Login in your Azure Management portal and navigate to your VM. Go to Endpoints and add an endpoint for port 9200 internal + external => ok => ok => ok. wait done!
partially taken from partial from http://blog.dimaj.net/content/howto-view-and-analyze-your-logs-web-page
cd ~
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
tar xzvf logstash-1.4.2.tar.gz
mv logstash-1.4.2 /opt/logstash
adduser --system --disabled-login --no-create-home --group logstash
usermod -a -G adm logstash
#auto start script for logstash
# download init script
wget -O /etc/init.d/logstash https://raw.githubusercontent.com/elasticsearch/logstash/master/pkg/logstash.sysv
# make it executable
chmod +x /etc/init.d/logstash
# make it auto-startable
update-rc.d logstash defaults
# configure logstash folder
# create logstash config dir
mkdir -p /etc/logstash/conf.d
# create logstash logs dir
mkdir /var/log/logstash
chown -R logstash: /var/log/logstash
# make home folder
mkdir /var/lib/logstash
chown -R logstash: /var/lib/logstash
Place your logstash config file (myconfig.conf) in /etc/logstash/conf.d, it will be automatically picked up after you start the logstash service (service logstash start)
Kibana 4 is no longer just fancy website but comes with a complete pre-packaged webserver.
cd ~
wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz
tar xzvf kibana-4.0.1-linux-x64.tar.gz
rm kibana-4.0.1-linux-x64.tar.gz
mv kibana-4.0.1-linux-x64 /var/opt/kibana
Before installing Kibana, let's set up a kibana user and group, which will own and run Kibana. If those commands fail because the 999 GID or UID already exist, replace the number with IDs that are free.
sudo groupadd -g 999 kibana
sudo useradd -u 999 -g 999 kibana
sudo chown -R kibana: /opt/kibana
vi /var/opt/kibana/config/kibana.yml
Kibana can be started by running /opt/kibana/bin/kibana, but we want it to run as a service. Download a Kibana init script with this command:
cd /etc/init.d && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-init
cd /etc/default && sudo curl -o kibana https://gist.githubusercontent.com/thisismitch/8b15ac909aed214ad04a/raw/fc5025c3fc499ad8262aff34ba7fde8c87ead7c0/kibana-4.x-default
Now enable the Kibana service, and start it:
sudo chmod +x /etc/init.d/kibana
sudo update-rc.d kibana defaults 96 9
sudo service kibana start
Curator deletes old time-based indices to prevent data clutter.
sudo apt-get install python-pip
sudo pip install elasticsearch-curator
#now get a crontab running
cat >> /etc/cron.d/elasticsearch_curator <<EOF
# delete everything older than 3 months old
@midnight root /usr/local/bin/curator delete indices --older-than 3 --time-unit months --timestring %Y.%m.%d --prefix logstash >> /var/log/elasticsearch-curator.log 2>&1
# remove old data above 180GB
@midnight root /usr/local/bin/curator delete --disk-space 180 indices --prefix logstash >> /var/log/elasticsearch-curator.log 2>&1
# optimize indices to merge segments
@midnight root /usr/local/bin/curator optimize --max_num_segments 2 indices --prefix logstash >> /var/log/elasticsearch-curator.log 2>&1
EOF