James Lewis james1ewis
- http://
james1ewis
/ gist:5047322
Created
February 27, 2013 11:41
the elasticsearch process showing the java environment variables that have been set.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /usr/lib/jvm/java-7-oracle/bin/java -Xms24g -Xmx24g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-0.20.4.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/srv/elasticsearch/logs -Des.default.path.data=/srv/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.ElasticSearch |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| rootLogger: INFO, console, file | |
| logger: | |
| # log action execution errors for easier debugging | |
| action: DEBUG | |
| # gateway | |
| gateway: DEBUG | |
| index.gateway: DEBUG | |
| # peer shard recovery |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| discovery.zen.ping.multicast.enabled: false | |
| discovery.zen.ping.unicast.hosts: ["10.XXX.X.XX", "10.XXX.X.XX"] #removed the IP address - assume they're correct | |
| discovery.zen.minimum_master_nodes: 2 | |
| cluster.name: prod-inbox-locker | |
| node.name: prod-inbox-elasticsearch01 | |
| index.number_of_shards: 100 | |
| index.number_of_replicas: 1 | |
| path.data: /srv/elasticsearch | |
| path.logs: /srv/elasticsearch/logs |
james1ewis
/ gist:5047202
Created
February 27, 2013 11:15
Logs from the node I sent the bulk index request to.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2013-02-26 17:14:53,006][WARN ][transport.netty ] [prod-inbox-elasticsearch00] Message not fully read (response) for [1192423] handler org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4@47582bc9, error [false], resetting | |
| [2013-02-26 17:14:53,088][WARN ][transport.netty ] [prod-inbox-elasticsearch00] Message not fully read (response) for [1192424] handler org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4@765a7846, error [false], resetting | |
| [2013-02-26 17:14:53,218][WARN ][transport.netty ] [prod-inbox-elasticsearch00] Message not fully read (response) for [1192425] handler org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4@2ef6aab5, error [false], resetting | |
| [2013-02-26 17:14:53,977][WARN ][transport.netty ] [prod-inbox-elasticsearch00] Message not fully read (response) for [1192427] handler org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4@dfbb050 |
james1ewis
/ gist:4318572
Created
December 17, 2012 14:13
The part of my elasticsearch startup script that shows the jvm environment variables I'm using.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Directory where the ElasticSearch binary distribution resides | |
| ES_HOME=/usr/share/$NAME | |
| # Heap Size (defaults to 256m min, 1g max) | |
| ES_HEAP_SIZE=6g | |
| # Heap new generation | |
| #ES_HEAP_NEWSIZE= | |
| # max direct memory |
james1ewis
/ gist:4318255
Created
December 17, 2012 13:24
Elasticsearch log output showing the final gc warnings up to the first out of memory exception. This now repeats in this loop: a few gc warnings, an out of memory exception, a few gc warnings, an out of memory exception etc.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2012-12-17 13:19:46,198][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6900][176] duration [20.8s], collections [1]/[21.4s], total [20.8s]/[41.6m], memory [5.9gb]->[5.9gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [2.1kb]->[19.2kb]/[16.6mb]}{[Survivor Space] [0b]->[0b]/[2mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]} | |
| [2012-12-17 13:20:07,810][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6901][177] duration [20.9s], collections [1]/[21.6s], total [20.9s]/[42m], memory [5.9gb]->[5.9gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [19.2kb]->[2.1kb]/[16.6mb]}{[Survivor Space] [0b]->[0b]/[2mb]}{[CMS Old Gen] [5.9gb]->[5.9gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]} | |
| [2012-12-17 13:20:41,924][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6902][179] duration [33.7s], collections [2]/[34.1s], total [33.7s]/[42.5m], memory [5.9gb]->[5. |
james1ewis
/ gist:4318237
Created
December 17, 2012 13:21
Copy of my elasticsearch.yml config. Note - mlockall does not work with the version of the OpenJDK that I'm running so should be disregarded.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| index.number_of_replicas: 0 | |
| bootstrap.mlockall: true | |
| http.max_content_length: 1000mb | |
james1ewis
/ gist:4318197
Created
December 17, 2012 13:15
Elasticsearch hot_threads output taken at the same time as my second elasticsearch log output.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| curl localhost:9200/_nodes/hot_threads | |
| ::: [Autolycus][8WVLJeoMQsGsGVcFDpSGOA][inet[/10.100.39.31:9300]] | |
| 12.0% (60ms out of 500ms) cpu usage by thread 'elasticsearch[Autolycus][flush][T#5]' | |
| 2/10 snapshots sharing following 22 elements | |
| org.apache.lucene.util.SorterTemplate.quickSort(SorterTemplate.java:117) | |
| org.apache.lucene.util.SorterTemplate.quickSort(SorterTemplate.java:118) | |
| org.apache.lucene.util.SorterTemplate.quickSort(SorterTemplate.java:118) | |
| org.apache.lucene.util.SorterTemplate.quickSort(SorterTemplate.java:68) | |
| org.apache.lucene.index.TermsHashPerField.sortPostings(TermsHashPerField.java:171) |
james1ewis
/ gist:4318193
Created
December 17, 2012 13:14
Elasticsearch hot_threads output taken at the same time as my first elasticsearch log output showing the gc warnings.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| curl localhost:9200/_nodes/hot_threads | |
| ::: [Autolycus][8WVLJeoMQsGsGVcFDpSGOA][inet[/10.100.39.31:9300]] | |
| 8.0% (40ms out of 500ms) cpu usage by thread 'elasticsearch[Autolycus][[inbox][4]: Lucene Merge Thread #265]' | |
| 5/10 snapshots sharing following 11 elements | |
| org.apache.lucene.store.DataOutput.writeVInt(DataOutput.java:82) | |
| org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:96) | |
| org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:548) | |
| org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:496) | |
| org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:428) |
james1ewis
/ gist:4318188
Created
December 17, 2012 13:13
Elasticsearch log output showing the ConcurrentMarkSweep gc warnings where the duration has increased. This duration increases linearly.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [2012-12-17 12:32:00,847][INFO ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6143][60] duration [8.4s], collections [1]/[8.7s], total [8.4s]/[5.9m], memory [4.4gb]->[4.2gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [5.2mb]->[2.1kb]/[16.6mb]}{[Survivor Space] [2mb]->[0b]/[2mb]}{[CMS Old Gen] [4.4gb]->[4.2gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]} | |
| [2012-12-17 12:32:32,417][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6162][61] duration [12.8s], collections [1]/[13.1s], total [12.8s]/[6.1m], memory [4.6gb]->[4.3gb]/[5.9gb], all_pools {[Code Cache] [6.8mb]->[6.8mb]/[48mb]}{[Eden Space] [15.4kb]->[13.6kb]/[16.6mb]}{[Survivor Space] [2mb]->[0b]/[2mb]}{[CMS Old Gen] [4.6gb]->[4.3gb]/[5.9gb]}{[CMS Perm Gen] [37.2mb]->[37.2mb]/[168mb]} | |
| [2012-12-17 12:33:03,645][WARN ][monitor.jvm ] [Autolycus] [gc][ConcurrentMarkSweep][6181][62] duration [12.6s], collections [1]/[13.1s], total [12.6s]/[6.3m], memory [4.8gb]->[4.4gb]/ |