setup elasticsearch cluster in production

elasticsearch.md

basic concept

• https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up
• https://www.elastic.co/blog/found-sizing-elasticsearch
• shard: https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticsearch-service-how-many-shards-do-i-need/
• https://www.codementor.io/ashish1dev/getting-started-with-elasticsearch-du107nett

docker

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.2.4

k8s

• https://github.com/pires/kubernetes-elasticsearch-cluster
• (TGIF) https://www.youtube.com/watch?v=8X8Rh_RKFSQ
• https://github.com/upmc-enterprises/elasticsearch-operator

cheatsheets

• http://elasticsearch-cheatsheet.jolicode.com
• https://github.com/dzharii/awesome-elasticsearch
• https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/

guide

• https://blog.insightdatascience.com/anatomy-of-an-elasticsearch-cluster-part-i-7ac9a13b05db
• https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-almost-definitive-guide-e614eabc1a87
• https://thoughts.t37.net/operating-large-elasticsearch-clusters-ab06670e2304
• https://thoughts.t37.net/migrating-a-130tb-cluster-from-elasticsearch-2-to-5-in-20-hours-with-0-downtime-and-a-rollback-39b4b4f29119
• https://www.elastic.co/guide/en/elasticsearch/reference/5.5/important-settings.html
• https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-ubuntu-14-04
• https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-centos-7
• https://www.elastic.co/blog/strings-are-dead-long-live-strings

index template

• https://jackhanington.com/blog/2014/12/11/create-a-custom-elasticsearch-template/
• https://www.elastic.co/blog/little-logstash-lessons-part-using-grok-mutate-type-data
• https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping
• https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
• uken/fluent-plugin-elasticsearch#33 (comment)

time series data

• https://cognitree.com/blog/time-series-data-management-elasticsearch/

rolling upgrade

• https://www.elastic.co/guide/en/elasticsearch/reference/5.6/rolling-upgrades.html

tuning

• https://tech.scrunch.com/blog/lessons-learned-from-a-year-of-running-elasticsearch-in-production/
• http://blogs.justenougharchitecture.com/running-elasticsearch-in-production/
• https://thehoard.blog/techniques-to-achieve-high-write-throughput-with-elasticsearch-519ead926b63

health

• _cluster/health?level=indices&pretty cluster health with indices

snapshot status

such as _snapshot/backup/snapshot_1/_status

checking restore status

/_cat/recovery?v /_recovery?human

aws

• https://aws.amazon.com/blogs/aws/connect-kinesis-to-elasticsearch/

community

• https://gitter.im/sksamuel/elastic4s