Elasticsearch configs

elasticsearch-for-write-heavy-cluster.yml

#
# Cluster and Node
# Note: We should always set the cluster name and nodes' name specifically.
#
cluster.name: elasticsearch_prod
node.name: elasticsearch_001
#node.(attribute): (attibute_value)

#
# Index
#
index.refresh_interval: 30s
index.translog.interval: 30s
# index.translog.flush_threshold_size: 1gb
index.gateway.local.sync: 30s
indices.memory.index_buffer_size: 35%
indices.memory.min_shard_index_buffer_size: 12mb
# If you are not using SSD, uncomment the following setting, which limits the number of concurrent merges to one
# index.merge.scheduler.max_thread_count: 1

#
# Paths
#
path.data: /mnt/elasticsearch/data
path.logs: /mnt/elasticsearch/logs
path.work: /mnt/elasticsearch/work

#
# Memory
#
bootstrap.mlockall: true

#
# Gateway (Presume you have 3 nodes in the cluster)
#
gateway.recover_after_nodes: 3
#gateway.expected_nodes: 3
#gatewat.recover_after_time: 5m

#
# Discovery 
#
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false 
discovery.zen.ping.unicast.hosts: ["host1", "host2", "host3:port3"]
# In a heavy-read/write environment, the machine might not be able to response the ping instantly.
# So we have to relax the ping interval a little.
discovery.zen.fd.ping_interval: 15s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5

# For aws-cloud plugin 
#cloud.aws.access_key: (your access key)
#cloud.aws.secret_key: (your secret key)
#cloud.aws.region: us-east
#discovery.type: ec2
#discovery.ec2.groups: (your security group ids/names)

#
# Actions
#
action.auto_create_index: false
action.disable_delete_all_indices: true

#
# Threadpool (the default settings is good enough)
# Note: A lot of posts state that we should enlarge the value of thread.xxx.size.
#       It's totally wrong! The effectiveness of threads depends on how many cores can do the jobs simultaneously.
#       If a very large threadpool size is specified, a lot of CPU time will be wasted on context-switching.
#
#threadpool.bulk.size: (# of CPU)*5
#threadpool.bulk.queue_size: -1

The indexing rate of a 3-node cluster with the above settings is 5k~7k docs/sec.
Node spec: AWS EC2 M3 xlarge instance
Doc size: ~500 bytes