Skip to content

Instantly share code, notes, and snippets.

@BlueStalker
Created March 8, 2013 10:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save BlueStalker/5115578 to your computer and use it in GitHub Desktop.
Save BlueStalker/5115578 to your computer and use it in GitHub Desktop.
Terms Facets with few hits got Elastic Out Of Memory Error.
Java Version: jdk1.7.0_05
ElasticSearch Engine Version: elasticsearch-0.20.5
Cluster Info: We have one master node, and 2 slave nodes, with all :
ES_HEAP_SIZE=3000M
ES_MIN_MEM=3000M
ES_MAX_MEM=3000M
On Master Node: ps aux | grep elas
root 1447 9.6 45.0 3840652 3679364 ? SLl Mar07 138:52 /root/jdk1.7.0_05/bin/java -Xms3000M -Xmx3000M -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=/root/elasticsearch-0.20.5 -cp :/root/elasticsearch-0.20.5/lib/elasticsearch-0.20.5.jar:/root/elasticsearch-0.20.5/lib/*:/root/elasticsearch-0.20.5/lib/sigar/* org.elasticsearch.bootstrap.ElasticSearch
Index name: surfikisemterms
the mapping is:
{"surfikisemterms":{"jdbc_search":{"properties":{"facets":{"dynamic":"true","properties":{"termscount":{"dynamic":"true","properties":{"terms":{"dynamic":"true","properties":{"field":{"type":"string"},"size":{"type":"long"}}}}}}},"from":{"type":"long"},"query":{"dynamic":"true","properties":{"bool":{"dynamic":"true","properties":{"must":{"dynamic":"true","properties":{"query_string":{"dynamic":"true","properties":{"query":{"type":"string"}}}}}}}}},"size":{"type":"long"}}},"jdbc":{"properties":{"keywords":{"type":"string","analyzer":"comma"},"time":{"type":"date","format":"dateOptionalTime"}}}}}
index setting is:
{"surfikisemterms":{"settings":{"index.analysis.tokenizer.commatokenizer.type":"pattern","index.analysis.analyzer.comma.type":"custom","index.number_of_replicas":"1","index.version.created":"200599","index.analysis.tokenizer.commatokenizer.pattern":",","index.analysis.analyzer.comma.tokenizer":"commatokenizer","index.number_of_shards":"5"}}}
So actually we have 2 fields in the surfikisemterms,
1. keywords, a list of string
2. time, the timestamp for these keywords
The query:
{
"query": {
"bool": {
"must": [
{
"range": {
"time": {
"from": "2013-03-07T11:29:19.000Z",
"to": "2013-03-07T11:30:19.000Z"
}
}
}
],
"must_not": [],
"should": []
}
},
"from": 0,
"size": 10,
"sort": [],
"facets": {}
}
Above query will return about 20 hits, with each hit the keyword list will no more than 20.
So, the terms facets according to above query:
Then if I do:
{
"query": {
"bool": {
"must": [
{
"range": {
"time": {
"from": "2013-03-07T11:29:19.000Z",
"to": "2013-03-07T11:30:19.000Z"
}
}
}
],
"must_not": [],
"should": []
}
},
"from": 0,
"size": 22,
"sort": [],
"facets": {
"termscount": {
"terms": {
"field": "keywords",
"size": 25
}
}
}
}
I can see following logs (on master node, name is Bobby Hutcherson) repeating for several time:
[2013-03-08 03:31:05,571][WARN ][index.cache.field.data.resident] [Surfiki
Master: Bobby Hutcherson] [surfikisemterms] loading field [keywords] caused
out of memory failure
java.lang.OutOfMemoryError: Java heap space..
Thanks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment