Notes to self while reviewing the PR for BEAM-5107.
ElasticsearchIOITcommon JDoc references mvn. To work around this quickly I did the following hack(!).
Added this to the elasticsearch-tests-common/build.gradle
task execute(type:JavaExec) {
main = mainClass
classpath = sourceSets.test.runtimeClasspath
}
Then hardcoded options.setElasticsearchServer("127.0.0.1"); in the ElasticsearchIOITCommon and then ran:
$ ./gradlew -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-common -PmainClass=org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOITCommon execute
I ran that each time I spun up an ES server for an IT test, verified that the indexes where as expected using http://localhost:9200/_cat/indices?v and then commented it back out as it stops the build run the IT.
testWriteVolumeWithFullAddressing failed with the following, but I highly suspect this is the thread_pool.bulk.queue_size so ignoring for now.
Caused by: java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO.checkForErrors(ElasticsearchIO.java:195)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1010)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.processElement(ElasticsearchIO.java:979)
Note that the instructions in create_elk_container for this is wrong as it spins up a ES2 stack and should be
sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it -e LOGSTASH_START=0 -e KIBANA_START=0 --name elk-5.4.3 sebp/elk:543
testWriteVolumeWithFullAddressing failed with the following which is related to an untuned thread_pool.bulk.queue_size so ignoring.
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
Document id 38022: rejected execution of org.elasticsearch.transport.TransportService$7@7881fbb2 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@62754059[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1241]] (es_rejected_execution_exception)
Document id 38023: rejected execution of org.elasticsearch.transport.TransportService$7@47aa1c88 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@62754059[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1241]] (es_rejected_execution_exception)
Document id 38024: rejected execution of org.elasticsearch.transport.TransportService$7@757034a4 on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@62754059[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1241]] (es_rejected_execution_exception)
Note that the instructions in create_elk_container for this is wrong as it spins up a ES2 stack. It should something like:
sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it -e LOGSTASH_START=0 -e KIBANA_START=0 --name elk-6.3.2 sebp/elk:632
testWriteVolumeWithFullAddressing failed with the following which is related to an untuned thread_pool.bulk.queue_size so ignoring.
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: Error writing to Elasticsearch, some elements could not be inserted:
Document id 31685: rejected execution of org.elasticsearch.transport.TransportService$7@634d63f0 on EsThreadPoolExecutor[name = i6l6CP9/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@60f85fa1[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1307]] (es_rejected_execution_exception)
Document id 31686: rejected execution of org.elasticsearch.transport.TransportService$7@36bead77 on EsThreadPoolExecutor[name = i6l6CP9/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@60f85fa1[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1307]] (es_rejected_execution_exception)
Document id 31687: rejected execution of org.elasticsearch.transport.TransportService$7@c64e4b5 on EsThreadPoolExecutor[name = i6l6CP9/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@60f85fa1[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1307]] (es_rejected_execution_exception)