Last active
August 29, 2015 14:04
-
-
Save furandon-pig/2669d7e32aab1c5f5eae to your computer and use it in GitHub Desktop.
OSvでelasticsearchを動作させる手順です。
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| OSvでelasticsearchを動作させる手順 | |
| ================================== | |
| OSvでelasticsearchを動作させる手順です。ここではデフォルトのOSvイメージを起動し、そこにelasticsearchのjarファイルを持ってゆく方法で動作させています。 | |
| elasticsearchのビルド | |
| --------------------- | |
| 最初にelasticsearchを自分の環境でビルドします。まずはelasticsearchをgit cloneします。 | |
| ``` | |
| [host_os]$ git clone https://github.com/elasticsearch/elasticsearch.git | |
| ``` | |
| pom.xmlを修正し、[SIGAR](https://github.com/hyperic/sigar "SIGAR")の利用を無効化します。SIGARを有効化していると、elasticsearchの起動時にlibsigar-amd64-linux.soをロードするため、OSv側で"Failed looking up symbol rpc_createerr"が発生してしまいます。 | |
| ``` | |
| --- pom.xml.orig 2014-07-15 11:29:42.227919714 +0900 | |
| +++ pom.xml 2014-07-15 11:30:02.800919365 +0900 | |
| @@ -295,7 +295,7 @@ | |
| <artifactId>sigar</artifactId> | |
| <version>1.6.4</version> | |
| <scope>compile</scope> | |
| - <optional>true</optional> | |
| + <optional>false</optional> | |
| </dependency> | |
| <!-- We don't use this since the publish pom is then messed up --> | |
| ``` | |
| elasticsearchをビルドします。ビルドが完了すると、targetディレクトリ以下にjarファイルが生成されます。 | |
| ``` | |
| [host_os]$ mvn clean package -DskipTests | |
| ``` | |
| 生成された以下のjarファイルをOSvにコピーします。 | |
| * target/elasticsearch-2.0.0-SNAPSHOT.jar | |
| * target/lib/*.jar | |
| OSvでのelasticsearchの起動 | |
| -------------------------- | |
| ### OSvの起動 | |
| OSvを起動します。ここではデフォルトのOSvイメージを使用します。NAT環境の場合は、TCPの9200番をポートフォワードさせてください。 | |
| ``` | |
| [host_os]$ ./scripts/run.py --forward tcp:9200::9200 | |
| ``` | |
| ### jarファイルのコピー | |
| ビルドしたelasticsearchのjarファイルをOSvにコピーします。 | |
| ``` | |
| [host_os]$ cd elasticsearch | |
| [host_os]$ scp -P 2222 target/lib/elasticsearch-2.0.0-SNAPSHOT.jar admin@localhost:~/ | |
| [host_os]$ scp -P 2222 target/lib/*.jar admin@localhost:~/ | |
| ``` | |
| elasticsearchのデータディレクトリ用にtmp/elasticsearchをmkdirしてから起動します。 | |
| ``` | |
| [/]% mkdir tmp/elasticsearch | |
| [/]% java -Djava.awt.headless=true -Des.foreground=yes -Des.path.home=/tmp/elasticsearch -cp /elasticsearch-2.0.0-SNAPSHOT.jar:/* org.elasticsearch.bootstrap.Elasticsearch | |
| ``` | |
| elasticsearchを起動すると以下のログが出力されます。SIGAR関連のロードで例外が発生しますが、起動処理は継続されます。 | |
| ``` | |
| [/]% log4j:WARN No appenders could be found for logger (node). | |
| log4j:WARN Please initialize the log4j system properly. | |
| log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. | |
| 23588 [Thread-11] DEBUG Sigar - no libsigar-amd64-linux.so in java.library.path | |
| org.hyperic.sigar.SigarException: no libsigar-amd64-linux.so in java.library.path | |
| at org.hyperic.sigar.Sigar.loadLibrary(Sigar.java:172) | |
| at org.hyperic.sigar.Sigar.<clinit>(Sigar.java:100) | |
| at org.elasticsearch.monitor.sigar.SigarService.<init>(SigarService.java:40) | |
| at org.elasticsearch.monitor.MonitorModule.configure(MonitorModule.java:70) | |
| at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60) | |
| at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204) | |
| at org.elasticsearch.common.inject.spi.Elements.getElements(Elements.java:85) | |
| at org.elasticsearch.common.inject.InjectorShell$Builder.build(InjectorShell.java:130) | |
| at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:99) | |
| at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93) | |
| at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) | |
| at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59) | |
| at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:193) | |
| at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159) | |
| at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70) | |
| at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:203) | |
| at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:606) | |
| at io.osv.ContextIsolator.runMain(ContextIsolator.java:230) | |
| at io.osv.ContextIsolator.access$400(ContextIsolator.java:32) | |
| at io.osv.ContextIsolator$3.run(ContextIsolator.java:118) | |
| 48261 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [generic], type [cached], keep_alive [30s] | |
| 48449 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [index], type [fixed], size [4], queue_size [200] | |
| 48519 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [bulk], type [fixed], size [4], queue_size [50] | |
| 48532 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [get], type [fixed], size [4], queue_size [1k] | |
| 48544 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [search], type [fixed], size [12], queue_size [1k] | |
| 48558 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [suggest], type [fixed], size [4], queue_size [1k] | |
| 48575 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [percolate], type [fixed], size [4], queue_size [1k] | |
| 48590 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m] | |
| 48623 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [flush], type [scaling], min [1], size [2], keep_alive [5m] | |
| 48637 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [merge], type [scaling], min [1], size [2], keep_alive [5m] | |
| 48650 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [refresh], type [scaling], min [1], size [2], keep_alive [5m] | |
| 48664 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [warmer], type [scaling], min [1], size [2], keep_alive [5m] | |
| 48683 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [snapshot], type [scaling], min [1], size [2], keep_alive [5m] | |
| 48702 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [snapshot_data], type [scaling], min [1], size [5], keep_alive [5m] | |
| 48716 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [optimize], type [fixed], size [1], queue_size [null] | |
| 48724 [Thread-11] DEBUG threadpool - [Copycat] creating thread_pool [bench], type [scaling], min [1], size [2], keep_alive [5m] | |
| 49167 [Thread-11] DEBUG monitor.jvm - [Copycat] enabled [true], last_gc_enabled [false], interval [1s], gc_threshold [{old=GcThreshold{name='old', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, default=GcThreshold{name='default', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, young=GcThreshold{name='young', warnThreshold=1000, infoThreshold=700, debugThreshold=400}}] | |
| 49224 [Thread-11] DEBUG monitor.os - [Copycat] Using probe [org.elasticsearch.monitor.os.JmxOsProbe@5397d562] with refresh_interval [1s] | |
| 49269 [Thread-11] DEBUG monitor.process - [Copycat] Using probe [org.elasticsearch.monitor.process.JmxProcessProbe@22503e0e] with refresh_interval [1s] | |
| 49454 [Thread-11] DEBUG monitor.jvm - [Copycat] Using refresh_interval [1s] | |
| 49466 [Thread-11] DEBUG monitor.network - [Copycat] Using probe [org.elasticsearch.monitor.network.JmxNetworkProbe@20a7b6c6] with refresh_interval [5s] | |
| 49548 [Thread-11] DEBUG monitor.network - [Copycat] net_info | |
| host [osv.local] | |
| eth0 display_name [eth0] | |
| address [/192.168.122.15] | |
| mtu [1500] multicast [true] ptp [false] loopback [false] up [true] virtual [false] | |
| lo0 display_name [lo0] | |
| address [/127.0.0.1] | |
| mtu [16384] multicast [true] ptp [false] loopback [true] up [true] virtual [false] | |
| 49626 [Thread-11] DEBUG monitor.fs - [Copycat] Using probe [org.elasticsearch.monitor.fs.JmxFsProbe@4a31c294] with refresh_interval [1s] | |
| 50269 [Thread-11] DEBUG transport.netty - [Copycat] using worker_count[8], port[9300-9400], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb] | |
| 51167 [Thread-11] DEBUG discovery.zen.ping.multicast - [Copycat] using group [224.2.2.4], with port [54328], ttl [3], and address [null] | |
| 51368 [Thread-11] DEBUG discovery.zen.ping.unicast - [Copycat] using initial hosts [], with concurrent_connects [10] | |
| 51433 [Thread-11] DEBUG discovery.zen - [Copycat] using ping.timeout [3s], join.timeout [1m], master_election.filter_client [true], master_election.filter_data [false] | |
| 51481 [Thread-11] DEBUG discovery.zen.elect - [Copycat] using minimum_master_nodes [-1] | |
| 51512 [Thread-11] DEBUG discovery.zen.fd - [Copycat] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3] | |
| 51628 [Thread-11] DEBUG discovery.zen.fd - [Copycat] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3] | |
| 68097 [Thread-11] DEBUG indices.store - [Copycat] using indices.store.throttle.type [MERGE], with index.store.throttle.max_bytes_per_sec [20mb] | |
| 69876 [Thread-11] DEBUG script - [Copycat] using script cache with max_size [500], expire [null] | |
| 70053 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4] | |
| 70089 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active] | |
| 70115 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using [cluster_concurrent_rebalance] with [2] | |
| 70265 [Thread-11] DEBUG gateway.local - [Copycat] using initial_shards [quorum], list_timeout [30s] | |
| 71759 [Thread-11] DEBUG indices.recovery - [Copycat] using max_bytes_per_sec[20mb], concurrent_streams [3], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true] | |
| 76509 [Thread-11] DEBUG http.netty - [Copycat] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[512kb->512kb] | |
| 76707 [Thread-11] DEBUG indices.memory - [Copycat] using index_buffer_size [175.3mb], with min_shard_index_buffer_size [4mb], max_shard_index_buffer_size [512mb], shard_inactive_time [30m] | |
| 76751 [Thread-11] DEBUG indices.cache.filter - [Copycat] using [node] weighted filter cache with size [10%], actual_size [175.3mb], expire [null], clean_interval [1m] | |
| 76911 [Thread-11] DEBUG indices.fielddata.cache - [Copycat] using size [-1] [-1b], expire [null] | |
| 77526 [Thread-11] DEBUG gateway.local.state.meta - [Copycat] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h] | |
| 78891 [Thread-11] DEBUG gateway.local.state.meta - [Copycat] took 1.3s to load state | |
| 78918 [Thread-11] DEBUG gateway.local.state.shards - [Copycat] took 2ms to load started shards state | |
| 79003 [Thread-11] DEBUG bulk.udp - [Copycat] using enabled [false], host [null], port [9700-9800], bulk_actions [1000], bulk_size [5mb], flush_interval [5s], concurrent_requests [4] | |
| 79113 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4] | |
| 79128 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active] | |
| 79147 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using [cluster_concurrent_rebalance] with [2] | |
| 79187 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4] | |
| 79206 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active] | |
| 79221 [Thread-11] DEBUG cluster.routing.allocation.decider - [Copycat] using [cluster_concurrent_rebalance] with [2] | |
| 79520 [Thread-11] INFO node - [Copycat] initialized | |
| 79529 [Thread-11] INFO node - [Copycat] starting ... | |
| 79970 [Thread-11] DEBUG netty.channel.socket.nio.SelectorUtil - Using select timeout of 500 | |
| 80072 [Thread-11] DEBUG netty.channel.socket.nio.SelectorUtil - Epoll-bug workaround enabled = false | |
| 82513 [Thread-11] DEBUG transport.netty - [Copycat] Bound to address [/0.0.0.0:9300] | |
| 82604 [Thread-11] INFO transport - [Copycat] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.122.15:9300]} | |
| 86605 [elasticsearch[Copycat][generic][T#1]] DEBUG discovery.zen - [Copycat] filtered ping responses: (filter_client[true], filter_data[false]) {none} | |
| 86846 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [zen-disco-join (elected_as_master)]: execute | |
| 86939 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] cluster state updated, version [1], source [zen-disco-join (elected_as_master)] | |
| 86996 [elasticsearch[Copycat][clusterService#updateTask][T#1]] INFO cluster.service - [Copycat] new_master [Copycat][qrq72x7mRl-hZjViIEZEMA][osv.local][inet[/192.168.122.15:9300]], reason: zen-disco-join (elected_as_master) | |
| 88554 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG transport.netty - [Copycat] connected to node [[Copycat][qrq72x7mRl-hZjViIEZEMA][osv.local][inet[/192.168.122.15:9300]]] | |
| 88616 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] publishing cluster state version 1 | |
| 88641 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] set local cluster state to version 1 | |
| 88719 [elasticsearch[Copycat][riverClusterService#updateTask][T#1]] DEBUG river.cluster - [Copycat] processing [reroute_rivers_node_changed]: execute | |
| 88757 [elasticsearch[Copycat][riverClusterService#updateTask][T#1]] DEBUG river.cluster - [Copycat] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| 88787 [Thread-11] INFO discovery - [Copycat] elasticsearch/qrq72x7mRl-hZjViIEZEMA | |
| 88772 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [zen-disco-join (elected_as_master)]: done applying updated cluster_state (version: 1) | |
| 89177 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [local-gateway-elected-state]: execute | |
| 89479 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] cluster state updated, version [2], source [local-gateway-elected-state] | |
| 89507 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] publishing cluster state version 2 | |
| 89601 [Thread-11] INFO http - [Copycat] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.122.15:9200]} | |
| 89640 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] set local cluster state to version 2 | |
| 89801 [elasticsearch[Copycat][riverClusterService#updateTask][T#1]] DEBUG river.cluster - [Copycat] processing [reroute_rivers_node_changed]: execute | |
| 89852 [elasticsearch[Copycat][riverClusterService#updateTask][T#1]] DEBUG river.cluster - [Copycat] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| 90193 [elasticsearch[Copycat][clusterService#updateTask][T#1]] INFO gateway - [Copycat] recovered [0] indices into cluster_state | |
| 90214 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [local-gateway-elected-state]: done applying updated cluster_state (version: 2) | |
| 90241 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [updating local node id]: execute | |
| 90256 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] cluster state updated, version [3], source [updating local node id] | |
| 90265 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] publishing cluster state version 3 | |
| 90282 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] set local cluster state to version 3 | |
| 90305 [elasticsearch[Copycat][riverClusterService#updateTask][T#1]] DEBUG river.cluster - [Copycat] processing [reroute_rivers_node_changed]: execute | |
| 90323 [elasticsearch[Copycat][riverClusterService#updateTask][T#1]] DEBUG river.cluster - [Copycat] processing [reroute_rivers_node_changed]: no change in cluster_state | |
| 90345 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [updating local node id]: done applying updated cluster_state (version: 3) | |
| 90351 [Thread-11] INFO node - [Copycat] started | |
| 98679 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [routing-table-updater]: execute | |
| 98713 [elasticsearch[Copycat][clusterService#updateTask][T#1]] DEBUG cluster.service - [Copycat] processing [routing-table-updater]: no change in cluster_state | |
| ``` | |
| 上記ログが出力された後、ホストOS側からクエリを投げてみます。以下のような応答が返ってきます。 | |
| ``` | |
| [host_os]$ curl -X GET http://localhost:9200/ | |
| { | |
| "status" : 200, | |
| "name" : "Copycat", | |
| "version" : { | |
| "number" : "2.0.0", | |
| "build_hash" : "ca194594b302f87922d0f3a479392bc6855b5400", | |
| "build_timestamp" : "2014-07-21T20:31:28Z", | |
| "build_snapshot" : true, | |
| "lucene_version" : "4.9" | |
| }, | |
| "tagline" : "You Know, for Search" | |
| } | |
| ``` | |
| ### elasticsearchのクエリ例 | |
| git cloneしたelasticsearch/README.textile内のサンプルを試してみます。 | |
| ``` | |
| [host_os]$ curl -XPUT 'http://localhost:9200/twitter/user/kimchy' -d '{ "name" : "Shay Banon" }' | |
| [host_os]$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d ' | |
| { | |
| "user": "kimchy", | |
| "postDate": "2009-11-15T13:12:00", | |
| "message": "Trying out Elasticsearch, so far so good?" | |
| }' | |
| [host_os]$ curl -XPUT 'http://localhost:9200/twitter/tweet/2' -d ' | |
| [host_os]$ curl -XGET 'http://localhost:9200/twitter/user/kimchy?pretty=true' | |
| { | |
| "_index" : "twitter", | |
| "_type" : "user", | |
| "_id" : "kimchy", | |
| "_version" : 1, | |
| "found" : true, | |
| "_source":{ "name" : "Shay Banon" } | |
| } | |
| [host_os]$ curl -XGET 'http://localhost:9200/twitter/tweet/1?pretty=true' | |
| { | |
| "_index" : "twitter", | |
| "_type" : "tweet", | |
| "_id" : "1", | |
| "_version" : 1, | |
| "found" : true, | |
| "_source": | |
| { | |
| "user": "kimchy", | |
| "postDate": "2009-11-15T13:12:00", | |
| "message": "Trying out Elasticsearch, so far so good?" | |
| } | |
| } | |
| [host_os]$ curl -XGET 'http://localhost:9200/twitter/tweet/2?pretty=true' | |
| { | |
| "_index" : "twitter", | |
| "_type" : "tweet", | |
| "_id" : "2", | |
| "_version" : 1, | |
| "found" : true, | |
| "_source": | |
| { | |
| "user": "kimchy", | |
| "postDate": "2009-11-15T14:12:12", | |
| "message": "Another tweet, will it be indexed?" | |
| } | |
| } | |
| ``` | |
| まだ確認できていない項目 | |
| ------------------------ | |
| まだ発生要因の切り分けは行えていませんが、elasticsearchを起動してしばらく経過すると、OSv側で以下のエラーが発生するようです。 | |
| ``` | |
| [/]% 4256315318470 < 4294967289570, now=4256315407620 | |
| [backtrace] | |
| 0x00000000003c621c <sched::timer_list::fired()+1100> | |
| 0x0000000000379e76 <interrupt_descriptor_table::invoke_interrupt(unsigned int)+166> | |
| 0x0000000000379f9b <interrupt+91> | |
| 0x0000000000378b6c <???+3640172> | |
| 0x00000000003c82bf <sched::cpu::idle()+31> | |
| 0x00000000003c82ee <???+3965678> | |
| 0x00000000003c84ea <thread_main_c+26> | |
| 0x0000000000378b95 <???+3640213> | |
| ``` |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment