Skip to content

Instantly share code, notes, and snippets.

@dattran-vn01

dattran-vn01/elasticsearch-pr-notes.md

Last active September 5, 2018 10:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dattran-vn01/f14e595ed765e0bdffdc2895271864ee to your computer and use it in GitHub Desktop.
Save dattran-vn01/f14e595ed765e0bdffdc2895271864ee to your computer and use it in GitHub Desktop.
Download ZIP
Integration test for branch: BEAM-5107-Support-ES-6.x-for-ElasticsearchIO
Raw
elasticsearch-pr-notes.md

Elasticsearch

Integration tests for the PR #6211 for BEAM-5107.

Maven instructions

ElasticsearchIOITcommon JDoc references mvn. To work around this quickly I did the following hack(!).

Added this to the elasticsearch-tests-common/build.gradle

task execute(type:JavaExec) {
  main = mainClass
  classpath = sourceSets.test.runtimeClasspath
}

Running the ES2 IT

Start ES2 server

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000 -p 9300:9300 -it -v $(pwd):/home/$USER/ --name elk-2.4  sebp/elk:es240_l240_k460

curl http://localhost:9200

{
  "name" : "Frankie and Victoria",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.4.0",
    "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",
    "build_timestamp" : "2016-08-29T09:14:17Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}

Prepare data

$ ./gradlew -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-common  -PmainClass=org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOITCommon execute

prepare-es2-data

Verify ES index

dattran@ubuntu:~$ curl http://localhost:9200/_cat/indices?v
health status index   pri rep docs.count docs.deleted store.size pri.store.size 
yellow open   .kibana   1   1          1            0      3.1kb          3.1kb 
yellow open   beam1     5   1      50000            0      2.1mb          2.1mb

verify-es2-index

Change thread pool bulk size

curl -X PUT -H "Content-Type: application/json" -d '{"transient": {"threadpool": {"bulk": {"queue_size": "200"}}}}' http://localhost:9200/_cluster/settings

IT test result

./gradlew integrationTest -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-2 -DintegrationTestPipelineOptions='["--elasticsearchServer=127.0.0.1","--elasticsearchHttpPort=9200"]' --tests org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOIT -DintegrationTestRunner=direct
Parallel execution is an incubating feature.
Parallel execution with configuration on demand is an incubating feature.

> Configure project :beam-model-pipeline
applyPortabilityNature with default configuration for project beam-model-pipeline

> Configure project :beam-model-fn-execution
applyPortabilityNature with default configuration for project beam-model-fn-execution

> Configure project :beam-model-job-management
applyPortabilityNature with default configuration for project beam-model-job-management

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3m 39s
58 actionable tasks: 1 executed, 57 up-to-date

es2-integration-test

Running the ES5 IT

Start ES5 server

# set max_map_count value (Linux)
sudo sysctl -w vm.max_map_count=262144

docker create -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000 -p 9300:9300 -it -v $(pwd):/home/$USER/ --name elk-5.4.3  sebp/elk:543

dattran@ubuntu:~$ curl http://localhost:9200
{
  "name" : "_ON7dR3",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "zPlss00yS62KHTPHMusUGg",
  "version" : {
    "number" : "5.4.3",
    "build_hash" : "eed30a8",
    "build_date" : "2017-06-22T00:34:03.743Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.1"
  },
  "tagline" : "You Know, for Search"
}

es5-curl

Prepare data

dattran@ubuntu:~/projects/beam$ ./gradlew -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-common  -PmainClass=org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOITCommon execute 
Parallel execution is an incubating feature.
Parallel execution with configuration on demand is an incubating feature.

> Configure project :beam-model-pipeline
applyPortabilityNature with default configuration for project beam-model-pipeline

> Configure project :beam-model-job-management
applyPortabilityNature with default configuration for project beam-model-job-management

> Configure project :beam-model-fn-execution
applyPortabilityNature with default configuration for project beam-model-fn-execution

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1m 1s
48 actionable tasks: 1 executed, 47 up-to-date

prepare-es5

Verify ES index

curl http://localhost:9200/_cat/indices?v
health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .kibana oQnOuBdEQnawkqFqfAqXIA   1   1          1            0      3.1kb          3.1kb
yellow open   beam1   Ex54i1A0R9SrNj1VmBP6lQ   5   1      50000            0      1.9mb          1.9mb

verify-es5

IT test result

dattran@ubuntu:~/projects/beam$ ./gradlew integrationTest -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-5 -DintegrationTestPipelineOptions='["--elasticsearchServer=127.0.0.1","--elasticsearchHttpPort=9200"]' --tests org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOIT -DintegrationTestRunner=direct
Parallel execution is an incubating feature.
Parallel execution with configuration on demand is an incubating feature.

> Configure project :beam-model-pipeline
applyPortabilityNature with default configuration for project beam-model-pipeline

> Configure project :beam-model-fn-execution
applyPortabilityNature with default configuration for project beam-model-fn-execution

> Configure project :beam-model-job-management
applyPortabilityNature with default configuration for project beam-model-job-management

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3m 33s
58 actionable tasks: 2 executed, 56 up-to-date

image

Running the ES6 IT

Start ES6 server

docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000 -p 9300:9300 -it -v $(pwd):/home/$USER/ --name elk-6.4.0  sebp/elk:640

dattran@ubuntu:~$ curl http://localhost:9200
{
  "name" : "aMwqeaS",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "19xIElVXRgSzhbblY8V5hQ",
  "version" : {
    "number" : "6.4.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "595516e",
    "build_date" : "2018-08-17T23:18:47.308994Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

image

Prepare data

dattran@ubuntu:~/projects/beam$ ./gradlew -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-common  -PmainClass=org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOITCommon execute 
Parallel execution is an incubating feature.
Parallel execution with configuration on demand is an incubating feature.

> Configure project :beam-model-pipeline
applyPortabilityNature with default configuration for project beam-model-pipeline

> Configure project :beam-model-job-management
applyPortabilityNature with default configuration for project beam-model-job-management

> Configure project :beam-model-fn-execution
applyPortabilityNature with default configuration for project beam-model-fn-execution

> Task :beam-sdks-java-io-elasticsearch-tests-common:execute
Sep 03, 2018 10:40:00 PM org.elasticsearch.client.RestClient logResponse
WARNING: request [POST http://elasticsearch-server-name:9200/beam1/test/_bulk?refresh=true] returned 1 warnings: [299 Elasticsearch-6.4.0-595516e "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template" "Tue, 04 Sep 2018 05:39:53 GMT"]

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 15s
48 actionable tasks: 1 executed, 47 up-to-date

image

Verify ES index

dattran@ubuntu:~/projects/beam$ curl http://localhost:9200/_cat/indices?v
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   beam1 webiBmHmRS-Ui0jq1COhJw   5   1      50000            0      1.4mb          1.4mb

image

IT test result

dattran@ubuntu:~/projects/beam$ ./gradlew integrationTest -p sdks/java/io/elasticsearch-tests/elasticsearch-tests-6 -DintegrationTestPipelineOptions='["--elasticsearchServer=127.0.0.1","--elasticsearchHttpPort=9200"]' --tests org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOIT -DintegrationTestRunner=direct
Starting a Gradle Daemon, 1 busy and 2 stopped Daemons could not be reused, use --status for details
Parallel execution is an incubating feature.
Parallel execution with configuration on demand is an incubating feature.

> Configure project :beam-model-pipeline
applyPortabilityNature with default configuration for project beam-model-pipeline

> Configure project :beam-model-fn-execution
applyPortabilityNature with default configuration for project beam-model-fn-execution

> Configure project :beam-model-job-management
applyPortabilityNature with default configuration for project beam-model-job-management

Deprecated Gradle features were used in this build, making it incompatible with Gradle 5.0.
See https://docs.gradle.org/4.8/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 4m 24s
58 actionable tasks: 2 executed, 56 up-to-date

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment