Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
ElasticsearchIOIT notes to self

Elasticsearch

Notes to self while reviewing the PR for BEAM-5107.

Maven instructions

ElasticsearchIOITcommon JDoc references mvn. To work around this quickly I did the following hack(!).

Added this to the elasticsearch-tests-common/build.gradle

task execute(type:JavaExec) {
  main = mainClass
  classpath = sourceSets.test.runtimeClasspath
}

Then hardcoded options.setElasticsearchServer("127.0.0.1"); in the ElasticsearchIOITCommon and then ran:

$ ./gradlew -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-common  -PmainClass=org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOITCommon execute 

I ran that each time I spun up an ES server for an IT test, verified that the indexes where as expected using http://localhost:9200/_cat/indices?v and then commented it back out as it stops the build run the IT.

Running the ES2 IT

testWriteVolumeWithFullAddressing failed with the following, but I highly suspect this is the thread_pool.bulk.queue_size so ignoring for now.

Caused by: java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
	at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:195)
	at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1010)
	at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.processElement(ElasticsearchIO.java:979)

Running the ES5 IT

Note that the instructions in create_elk_container for this is wrong as it spins up a ES2 stack and should be

sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it -e LOGSTASH_START=0 -e KIBANA_START=0 --name elk-5.4.3 sebp/elk:543

testWriteVolumeWithFullAddressing failed with the following which is related to an untuned thread_pool.bulk.queue_size so ignoring.

org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
Document id 38022: rejected execution of org.elasticsearch.transport.TransportService$7@7881fbb2 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@62754059[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1241]] (es_rejected_execution_exception)
Document id 38023: rejected execution of org.elasticsearch.transport.TransportService$7@47aa1c88 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@62754059[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1241]] (es_rejected_execution_exception)
Document id 38024: rejected execution of org.elasticsearch.transport.TransportService$7@757034a4 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@62754059[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1241]] (es_rejected_execution_exception)

Running the ES6 IT

Note that the instructions in create_elk_container for this is wrong as it spins up a ES2 stack. It should something like:

sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it -e LOGSTASH_START=0 -e KIBANA_START=0 --name elk-6.3.2 sebp/elk:632

testWriteVolumeWithFullAddressing failed with the following which is related to an untuned thread_pool.bulk.queue_size so ignoring.

org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
Document id 31685: rejected execution of org.elasticsearch.transport.TransportService$7@634d63f0 on EsThreadPoolExecutor[name = i6l6CP9/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@60f85fa1[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1307]] (es_rejected_execution_exception)
Document id 31686: rejected execution of org.elasticsearch.transport.TransportService$7@36bead77 on EsThreadPoolExecutor[name = i6l6CP9/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@60f85fa1[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1307]] (es_rejected_execution_exception)
Document id 31687: rejected execution of org.elasticsearch.transport.TransportService$7@c64e4b5 on EsThreadPoolExecutor[name = i6l6CP9/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@60f85fa1[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1307]] (es_rejected_execution_exception)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.