Skip to content

Instantly share code, notes, and snippets.

@apolloclark

apolloclark/elasticsearch_cheatsheet.md

Last active September 5, 2021 19:42
Show Gist options
  • Save apolloclark/c9eb0c1a01798ac2e48492ceeb367a4f to your computer and use it in GitHub Desktop.
Save apolloclark/c9eb0c1a01798ac2e48492ceeb367a4f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
elasticsearch_cheatsheet.md

Elasticsearch Cheatsheet

This is a collection of the most common commands I run while administering Elasticsearch databases. The variables shown between the open and closed tags, "<" and ">", should be replaced with a name you choose.

https://www.elastic.co/guide/en/elasticsearch/client/curator/5.x/command-line.html

I would suggest using my ELK vagrant project to use these commands:

https://github.com/apolloclark/elk


Notes

  • default cluster name is "elasticsearch"
  • default, each index has 5 primary shards, 1 replica
  • better to over-provision shards initially, ~4 shards-per-node is good
  • scale out nodes horizontally over time, until it's one-to-one with nodes
  • Lucene can address up to 2 billion documents (2^31 - 128)
  • mappings are essentially flattened into a single, global schema for the entire index
  • doc values often reserve a fixed amount of disk space for every document
  • shard data size should be balanced between nodes
  • a single slow node will slow down the entire cluster
  • version 5.x added "text" and "keyword" data types, replacing "string"
  • lenient boolean is deprecated
  • analyzer = filter, tokenizer, token filter
  • max content legnth default = 100 MB
  • max URL length default = 4 KB
  • max header size default = 8 KB


Setup

clear the screen
(CTRL + L)
configure
/etc/elasticsearch/elasticsearch.yml
debug logs
/var/log/elasticsearch/elasticsearch.log
/var/log/elasticsearch/elasticsearch.log.*
/var/log/elasticsearch/elasticsearch_deprecation.log
/var/log/elasticsearch/elasticsearch_index_search_slowlog.log
/var/log/elasticsearch/elasticsearch_index_indexing_slowlog.log.log
formatting

https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html

# print output with indentation
?pretty=true

# print output in YAML format
?format=yaml

# print units with human readable units
?human=<true | false>

# print ouput flattened
?flat_settings=<true | false>

# explain more details about query execution
?explain

# include detailed debug output on failure
&error_trace=true

# filter to only receive specific fields
&filter_path=<field>
&filter_path=<field>,<field>
&filter_path=<field>.<field>
&filter_path=<field>.*
&filter_path=<field>.**
&filter_path=-<field>




Configuration

# disable selecting indexes within the request body
rest.action.multi.allow_explicit_index: false

# disable automatic index creation
action.auto_create_index: false

# set automatic index creation
action.auto_create_index: <+aaa*,-bbb*,+ccc*,-*>

# disable automatic mapping
index.mapper.dynamic: false

# configure number of shards to search
action.search.shard_count.limit

# set search timeout limit
search.default_search_timeout

# enable fast search cancellation
search.low_level_cancellation: true




Cluster

show version
curl -s -XGET 'http://127.0.0.1:9200?filter_path=version.number&pretty=false' | \
  awk -F'"' {'print $6'}
list cluster name
curl -s -XGET 'http://127.0.0.1:9200?filter_path=cluster_name&pretty=false' | \
  awk -F'"' {'print $4'}
show cluster health
curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty'
show cluster state
curl -XGET 'http://127.0.0.1:9200/_cluster/state?pretty'
show cluster metric stats
curl -XGET 'http://127.0.0.1:9200/_cluster/stats?pretty'
show cluster settings
curl -XGET 'http://127.0.0.1:9200/_cluster/settings?pretty'
show all cluster settings
curl -XGET 'http://127.0.0.1:9200/_settings?pretty'




Indices

https://www.elastic.co/blog/index-vs-type

all indices
curl -XGET 'http://127.0.0.1:9200/_cat/indices?v'
all indices monitoring
curl -XGET 'http://127.0.0.1:9200/_all/_stats?pretty'
curl -XGET 'http://127.0.0.1:9200/_all/_segments?pretty'
curl -XGET 'http://127.0.0.1:9200/_all/_recovery?pretty&human'
single index monitoring
curl -XGET 'http://127.0.0.1:9200/<index>/_stats?pretty'
curl -XGET 'http://127.0.0.1:9200/<index>/_segments?pretty'
curl -XGET 'http://127.0.0.1:9200/<index>/_recovery?pretty&human'
indices management
POST /<index>/_cache/clear
POST /<index>/_refresh
POST /<index>/_flush
POST /<index>/_forcemerge
POST /<index>/_upgrade
GET /<index>/_upgrade?pretty&human
create index
curl -XPUT 'http://127.0.0.1:9200/<index>?pretty'

curl -XPUT 'http://127.0.0.1:9200/twitter?pretty'
one index, all details
curl -XGET 'http://127.0.0.1:9200/<index>?pretty'

curl -XGET 'http://127.0.0.1:9200/twitter?pretty'

curl -XGET 'http://127.0.0.1:9200/.kibana?pretty'
one index, alias
curl -XGET 'http://127.0.0.1:9200/<index>/_alias?pretty'

curl -XGET 'http://127.0.0.1:9200/twitter/_alias?pretty'

curl -XGET 'http://127.0.0.1:9200/.kibana/_alias?pretty'
one index, settings
curl -XGET 'http://127.0.0.1:9200/<index>/_settings?pretty'

curl -XGET 'http://127.0.0.1:9200/twitter/_settings?pretty'

curl -XGET 'http://127.0.0.1:9200/.kibana/_settings?pretty'
delete one index
curl -XDELETE 'http://127.0.0.1:9200/<index>?pretty'

curl -XDELETE 'http://127.0.0.1:9200/twitter?pretty'




Mappings

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

field datatypes

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html

logic = boolean
number = long, integer, short, byye, double, float, half_float, scaled_float
ranges = interger_range, float_range, long_range, double_range, date_range
string = text, keyword
dates = date
data = binary, array, object, nested
geo = geo_point, geo_shape
special = ip, completion, token_count, murmur3, attachment
list all indices, all mappings
curl -XGET 'http://127.0.0.1:9200/_mapping?pretty'
list one index, all mappings
curl -XGET 'http://127.0.0.1:9200/<index>/_mapping?pretty'

curl -XGET 'http://127.0.0.1:9200/twitter/_mapping?pretty'

curl -XGET 'http://127.0.0.1:9200/.kibana/_mapping?pretty'




Types

list all indices, all types
curl -s -XGET 'http://127.0.0.1:9200/_mapping' | \
  jq 'to_entries | .[] | {(.key): .value.mappings | keys}'
list one index, all types
curl -s -XGET 'http://127.0.0.1:9200/<index>/_mapping' | \
  jq '.[].mappings[] | keys'

curl -s -XGET 'http://127.0.0.1:9200/twitter/_mapping' | \
  jq '.[].mappings | keys'

curl -s -XGET 'http://127.0.0.1:9200/.kibana/_mapping' | \
  jq '.[].mappings | keys'

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_mapping' | \
  jq '.[].mappings | keys'

curl -s -XGET 'http://127.0.0.1:9200/metricbeat-*/_mapping' | \
  jq '.[].mappings | keys'

curl -s -XGET 'http://127.0.0.1:9200/heartbeat-*/_mapping' | \
  jq '.[].mappings | keys'
list one index, one type, all mappings
curl -XGET 'http://127.0.0.1:9200/<index>/_mapping/<type>'

curl -XGET 'http://127.0.0.1:9200/twitter/_mapping/tweet?pretty'

curl -XGET 'http://127.0.0.1:9200/.kibana/_mapping/index-pattern?pretty'

curl -XGET 'http://127.0.0.1:9200/filebeat-*/_mapping/authlog?pretty'
list one index, all types, all mappings, collapsed to type
curl -s -XGET 'http://127.0.0.1:9200/<index>/_mapping' | \
  jq '. |= .[].mappings' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'

curl -s -XGET 'http://127.0.0.1:9200/.kibana/_mapping' | \
  jq '. |= .[].mappings' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_mapping' | \
  jq '. |= .[].mappings' | \
  jq 'del(.[].properties.type)' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'

curl -s -XGET 'http://127.0.0.1:9200/metricbeat-*/_mapping' | \
  jq '. |= .[].mappings' | \
  jq 'del(.[].properties.type)' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'

curl -s -XGET 'http://127.0.0.1:9200/heartbeat-*/_mapping' | \
  jq '. |= .[].mappings' | \
  jq 'del(.[].properties.type)' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'
list one index, one type, all mappings, collapsed to type
curl -XGET 'http://127.0.0.1:9200/<index>/_mapping/<type>?pretty'

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_mapping/authlog' | \
  jq '.[].mappings[]' | \
  jq 'del(.[].type)' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_mapping/syslog' | \
  jq '.[].mappings[]' | \
  jq 'del(.[].type)' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_mapping/misclog' | \
  jq '.[].mappings[]' | \
  jq 'del(.[].type)' | \
  jq 'walk( if type=="object" and has("properties") then . |= .properties else . end )' | \
  jq 'walk( if type=="object" and has("type") then . |= .type else . end )'




Nodes

list nodes
curl -XGET 'http://127.0.0.1:9200/_cat/nodes?v'
show node status
curl -XGET 'http://127.0.0.1:9200/_nodes/stats?pretty'




Shards

list shards
curl -XGET 'http://127.0.0.1:9200/_cat/shards?v'




Documents

insert data
curl -XPOST 'http://127.0.0.1:9200/twitter/tweet/?pretty' \
  -H 'Content-Type: application/json' \
-d'
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}
'

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html

search all documents, all indices
curl -XGET 'http://127.0.0.1:9200/_all/_search?pretty=true&q=*:*'

curl -s -XGET 'http://127.0.0.1:9200/_all/_search?pretty=true&q=*:*' |\
  jq '.hits.hits'
search all documents, one index
curl -XGET 'http://127.0.0.1:9200/<index>/_search?pretty=true&q=*:*'

curl -s -XGET 'http://127.0.0.1:9200/twitter/_search?pretty=true&q=*:*' |\
  jq '.hits.hits[]._source'
  
curl -s -XGET 'http://127.0.0.1:9200/.kibana/_search?pretty=true&q=*:*' |\
  jq '.hits.hits[]._source'

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_search?pretty=true&q=*:*' |\
  jq '.hits.hits[]._source'

curl -s -XGET 'http://127.0.0.1:9200/metricbeat-*/_search?pretty=true&q=*:*' |\
  jq '.hits.hits[]._source'

curl -s -XGET 'http://127.0.0.1:9200/heartbeat-*/_search?pretty=true&q=*:*' |\
  jq '.hits.hits[]._source'
search all documents, one index, one type
curl -XGET 'http://127.0.0.1:9200/<index>/<type>/_search?pretty=true&q=*:*'

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/authlog/_search?pretty=true&q=*:*' |\
  jq '.hits.hits[]._source'
search filebeat documents, count results

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html

curl -s -XGET -G 'http://127.0.0.1:9200/filebeat-*/_search' \
    -d 'q=message:install' \
    -d 'size=0' \
    -d 'terminate_after=1' \
    -d 'pretty' | \
  jq '.hits.total'
search filebeat documents, get only the message field

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-collapse.html

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_search?pretty=true&q=message:install' | \
  jq '[.hits.hits[]._source.message]'

curl -s -XGET -G 'http://127.0.0.1:9200/filebeat-*/misclog/_search' \
    -d '_source=message' \
    -d 'filter_path=hits.hits._source.message' \
    -d 'pretty' | \
  jq '[.hits.hits[]._source.message]'
search filebeat documents, for installation, get only the message field

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-uri-request.html https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-collapse.html

curl -s -XGET 'http://127.0.0.1:9200/filebeat-*/_search?pretty=true&q=message:install' | \
  jq '[.hits.hits[]._source.message]'

curl -s -XGET -G 'http://127.0.0.1:9200/filebeat-*/misclog/_search' \
    -d 'q=message:install' \
    -d '_source=message' \
    -d 'filter_path=hits.hits._source.message' \
    -d 'pretty' | \
  jq '[.hits.hits[]._source.message]'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment