Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@wingchen

wingchen/gist:7eb26cffbe59f6626e5a Secret

Last active August 29, 2015 14:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wingchen/7eb26cffbe59f6626e5a to your computer and use it in GitHub Desktop.
Save wingchen/7eb26cffbe59f6626e5a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/02/09 15:34:29 INFO SparkContext: Running Spark version 1.3.0-SNAPSHOT
15/02/09 15:34:29 INFO SecurityManager: Changing view acls to: wchen
15/02/09 15:34:29 INFO SecurityManager: Changing modify acls to: wchen
15/02/09 15:34:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(wchen); users with modify permissions: Set(wchen)
15/02/09 15:34:30 INFO Slf4jLogger: Slf4jLogger started
15/02/09 15:34:30 INFO Remoting: Starting remoting
15/02/09 15:34:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@localhost:62058]
15/02/09 15:34:30 INFO Utils: Successfully started service 'sparkDriver' on port 62058.
15/02/09 15:34:30 INFO SparkEnv: Registering MapOutputTracker
15/02/09 15:34:30 INFO SparkEnv: Registering BlockManagerMaster
15/02/09 15:34:30 INFO DiskBlockManager: Created local directory at /var/folders/_c/gf3rvwm571q3cgxzw8rjwhmw0000gp/T/spark-local-20150209153430-8801
15/02/09 15:34:30 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/02/09 15:34:30 INFO HttpFileServer: HTTP File server directory is /var/folders/_c/gf3rvwm571q3cgxzw8rjwhmw0000gp/T/spark-dad37ed1-027e-44e1-889a-657f15276e71
15/02/09 15:34:30 INFO HttpServer: Starting HTTP Server
15/02/09 15:34:30 INFO Utils: Successfully started service 'HTTP file server' on port 62059.
15/02/09 15:34:30 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/02/09 15:34:30 INFO SparkUI: Started SparkUI at http://localhost:4040
15/02/09 15:34:30 INFO SparkContext: Added JAR file:/Users/wchen/spark/elasticsearch-hadoop/build/libs/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar at http://127.0.0.1:62059/jars/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT.jar with timestamp 1423524870776
15/02/09 15:34:30 INFO SparkContext: Added JAR file:/Users/wchen/spark/es_reader/target/es-reader-1.0.jar at http://127.0.0.1:62059/jars/es-reader-1.0.jar with timestamp 1423524870777
15/02/09 15:34:30 INFO AkkaUtils: Connecting to HeartbeatReceiver: akka.tcp://sparkDriver@localhost:62058/user/HeartbeatReceiver
15/02/09 15:34:30 INFO NettyBlockTransferService: Server created on 62060
15/02/09 15:34:30 INFO BlockManagerMaster: Trying to register BlockManager
15/02/09 15:34:30 INFO BlockManagerMasterActor: Registering block manager localhost:62060 with 265.4 MB RAM, BlockManagerId(<driver>, localhost, 62060)
15/02/09 15:34:30 INFO BlockManagerMaster: Registered BlockManager
15/02/09 15:34:31 TRACE CommonsHttpTransport: Opening HTTP transport to develop.es.puppy.int.mycom.com:9200
15/02/09 15:34:31 TRACE CommonsHttpTransport: Tx [GET]@[develop.es.puppy.int.mycom.com:9200][_nodes/transport] w/ payload [null]
15/02/09 15:34:31 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"cluster_name":"nyuzu","nodes":{"JRpaF5f7T2uLfJ90-sLBLA":{"name":"es-client001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.68:9300]","hostname":"es-client001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","http_address":"inet[/10.223.128.68:9500]","attributes":{"data":"false","max_local_storage_nodes":"1","master":"false"},"transport":{"bound_address":"inet[/0.0.0.0:9300]","publish_address":"inet[/10.223.128.68:9300]"}},"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","hostname":"es-data002.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"},"transport":{"bound_address":"inet[/0.0.0.0:9300]","publish_address":"inet[/10.223.128.69:9300]"}},"4d63O0bTSxyoVLE7DNDuiA":{"name":"es-master01.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.209.163.171:9300]","hostname":"es-master01.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"data":"false","max_local_storage_nodes":"1","master":"true"},"transport":{"bound_address":"inet[/0.0.0.0:9300]","publish_address":"inet[/10.209.163.171:9300]"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","hostname":"es-data001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"},"transport":{"bound_address":"inet[/0.0.0.0:9300]","publish_address":"inet[/10.223.128.70:9300]"}}}}]
15/02/09 15:34:31 DEBUG JavaEsRDD: Nodes discovery enabled - found [10.223.128.68:9500]
15/02/09 15:34:31 TRACE CommonsHttpTransport: Closing HTTP transport to develop.es.puppy.int.mycom.com:9200
15/02/09 15:34:31 TRACE CommonsHttpTransport: Opening HTTP transport to 10.223.128.68:9500
15/02/09 15:34:31 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][_nodes/http] w/ payload [null]
15/02/09 15:34:31 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"cluster_name":"nyuzu","nodes":{"JRpaF5f7T2uLfJ90-sLBLA":{"name":"es-client001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.68:9300]","hostname":"es-client001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","http_address":"inet[/10.223.128.68:9500]","attributes":{"data":"false","max_local_storage_nodes":"1","master":"false"},"http":{"bound_address":"inet[/0.0.0.0:9500]","publish_address":"inet[/10.223.128.68:9500]","max_content_length":"100mb","max_content_length_in_bytes":104857600}},"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","hostname":"es-data002.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}},"4d63O0bTSxyoVLE7DNDuiA":{"name":"es-master01.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.209.163.171:9300]","hostname":"es-master01.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"data":"false","max_local_storage_nodes":"1","master":"true"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","hostname":"es-data001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}}}}]
15/02/09 15:34:31 DEBUG JavaEsRDD: Found client nodes [10.223.128.68:9500]
15/02/09 15:34:31 DEBUG JavaEsRDD: Filtered discovered only nodes [develop.es.puppy.int.mycom.com:9200, 10.223.128.68:9500] to client-only [10.223.128.68:9500]
15/02/09 15:34:31 TRACE CommonsHttpTransport: Closing HTTP transport to 10.223.128.68:9500
15/02/09 15:34:31 TRACE CommonsHttpTransport: Opening HTTP transport to 10.223.128.68:9500
15/02/09 15:34:31 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{
"ok" : true,
"status" : 200,
"name" : "es-client001.puppy.static.ord.develop.mycomops.com",
"version" : {
"number" : "0.90.5",
"build_hash" : "c8714e8e0620b62638f660f6144831792b9dedee",
"build_timestamp" : "2013-09-17T12:50:20Z",
"build_snapshot" : false,
"lucene_version" : "4.4"
},
"tagline" : "You Know, for Search"
}]
15/02/09 15:34:32 DEBUG JavaEsRDD: Discovered Elasticsearch version [0.90.5]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Closing HTTP transport to 10.223.128.68:9500
15/02/09 15:34:32 TRACE CommonsHttpTransport: Opening HTTP transport to 10.223.128.68:9500
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [HEAD]@[10.223.128.68:9500][mo-sn-2013-08/searchable-news-article] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][mo-sn-2013-08/_search_shards] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"nodes":{"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","attributes":{"max_local_storage_nodes":"1","master":"false"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","attributes":{"max_local_storage_nodes":"1","master":"false"}}},"shards":[[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":16,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":16,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":17,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":17,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":14,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":14,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":15,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":15,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":11,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":11,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":10,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":10,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":13,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":13,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":12,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":12,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":7,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":7,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":6,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":6,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":9,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":9,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":8,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":8,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":3,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":3,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":2,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":2,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":5,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":5,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":4,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":4,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":1,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":1,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":0,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":0,"index":"mo-sn-2013-08"}]]}]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][_nodes/http] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"cluster_name":"nyuzu","nodes":{"JRpaF5f7T2uLfJ90-sLBLA":{"name":"es-client001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.68:9300]","hostname":"es-client001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","http_address":"inet[/10.223.128.68:9500]","attributes":{"data":"false","max_local_storage_nodes":"1","master":"false"},"http":{"bound_address":"inet[/0.0.0.0:9500]","publish_address":"inet[/10.223.128.68:9500]","max_content_length":"100mb","max_content_length_in_bytes":104857600}},"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","hostname":"es-data002.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}},"4d63O0bTSxyoVLE7DNDuiA":{"name":"es-master01.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.209.163.171:9300]","hostname":"es-master01.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"data":"false","max_local_storage_nodes":"1","master":"true"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","hostname":"es-data001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}}}}]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][mo-sn-2013-08/_aliases] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"mo-sn-2013-08":{"aliases":{"news":{}}}}]
15/02/09 15:34:32 WARN RestRepository: Cannot find node with id [gy_92kljRluUFT9N022KDw] (is HTTP enabled?) from shard [Shard[state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, name=16, index=mo-sn-2013-08]] in nodes [{JRpaF5f7T2uLfJ90-sLBLA=Node[id=JRpaF5f7T2uLfJ90-sLBLA, name=es-client001.nyuzu.static.ord.develop.mycomops.com, ipAddress=10.223.128.68, httpPort=9500]}]; layout [[[{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=16, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=16, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=17, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=17, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=14, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=14, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=15, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=15, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=11, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=11, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=10, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=10, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=13, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=13, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=12, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=12, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=7, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=7, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=6, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=6, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=9, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=9, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=8, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=8, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=3, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=3, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=2, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=2, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=5, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=5, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=4, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=4, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=1, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=1, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=0, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=0, index=mo-sn-2013-08}]]]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][mo-sn-2013-08/_search_shards] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"nodes":{"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","attributes":{"max_local_storage_nodes":"1","master":"false"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","attributes":{"max_local_storage_nodes":"1","master":"false"}}},"shards":[[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":16,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":16,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":17,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":17,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":14,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":14,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":15,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":15,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":11,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":11,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":10,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":10,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":13,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":13,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":12,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":12,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":7,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":7,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":6,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":6,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":9,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":9,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":8,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":8,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":3,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":3,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":2,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":2,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":5,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":5,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":4,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":4,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":1,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":1,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":0,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":0,"index":"mo-sn-2013-08"}]]}]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][_nodes/http] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"cluster_name":"nyuzu","nodes":{"JRpaF5f7T2uLfJ90-sLBLA":{"name":"es-client001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.68:9300]","hostname":"es-client001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","http_address":"inet[/10.223.128.68:9500]","attributes":{"data":"false","max_local_storage_nodes":"1","master":"false"},"http":{"bound_address":"inet[/0.0.0.0:9500]","publish_address":"inet[/10.223.128.68:9500]","max_content_length":"100mb","max_content_length_in_bytes":104857600}},"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","hostname":"es-data002.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}},"4d63O0bTSxyoVLE7DNDuiA":{"name":"es-master01.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.209.163.171:9300]","hostname":"es-master01.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"data":"false","max_local_storage_nodes":"1","master":"true"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","hostname":"es-data001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}}}}]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][mo-sn-2013-08/_aliases] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"mo-sn-2013-08":{"aliases":{"news":{}}}}]
15/02/09 15:34:32 WARN RestRepository: Cannot find node with id [Q_5mAcbBS_KGm6dEVp3oqQ] (is HTTP enabled?) from shard [Shard[state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, name=16, index=mo-sn-2013-08]] in nodes [{JRpaF5f7T2uLfJ90-sLBLA=Node[id=JRpaF5f7T2uLfJ90-sLBLA, name=es-client001.nyuzu.static.ord.develop.mycomops.com, ipAddress=10.223.128.68, httpPort=9500]}]; layout [[[{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=16, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=16, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=17, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=17, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=14, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=14, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=15, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=15, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=11, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=11, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=10, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=10, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=13, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=13, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=12, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=12, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=7, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=7, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=6, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=6, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=9, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=9, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=8, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=8, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=3, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=3, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=2, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=2, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=5, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=5, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=4, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=4, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=1, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=1, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=0, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=0, index=mo-sn-2013-08}]]]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][mo-sn-2013-08/_search_shards] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"nodes":{"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","attributes":{"max_local_storage_nodes":"1","master":"false"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","attributes":{"max_local_storage_nodes":"1","master":"false"}}},"shards":[[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":16,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":16,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":17,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":17,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":14,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":14,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":15,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":15,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":11,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":11,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":10,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":10,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":13,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":13,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":12,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":12,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":7,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":7,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":6,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":6,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":9,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":9,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":8,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":8,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":3,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":3,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":2,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":2,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":5,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":5,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":4,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":4,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":1,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":1,"index":"mo-sn-2013-08"}],[{"state":"STARTED","primary":false,"node":"Q_5mAcbBS_KGm6dEVp3oqQ","relocating_node":null,"shard":0,"index":"mo-sn-2013-08"},{"state":"STARTED","primary":true,"node":"gy_92kljRluUFT9N022KDw","relocating_node":null,"shard":0,"index":"mo-sn-2013-08"}]]}]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][_nodes/http] w/ payload [null]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"ok":true,"cluster_name":"nyuzu","nodes":{"JRpaF5f7T2uLfJ90-sLBLA":{"name":"es-client001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.68:9300]","hostname":"es-client001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","http_address":"inet[/10.223.128.68:9500]","attributes":{"data":"false","max_local_storage_nodes":"1","master":"false"},"http":{"bound_address":"inet[/0.0.0.0:9500]","publish_address":"inet[/10.223.128.68:9500]","max_content_length":"100mb","max_content_length_in_bytes":104857600}},"Q_5mAcbBS_KGm6dEVp3oqQ":{"name":"es-data002.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.69:9300]","hostname":"es-data002.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}},"4d63O0bTSxyoVLE7DNDuiA":{"name":"es-master01.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.209.163.171:9300]","hostname":"es-master01.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"data":"false","max_local_storage_nodes":"1","master":"true"}},"gy_92kljRluUFT9N022KDw":{"name":"es-data001.nyuzu.static.ord.develop.mycomops.com","transport_address":"inet[/10.223.128.70:9300]","hostname":"es-data001.nyuzu.static.ord.develop.mycomops.com","version":"0.90.5","attributes":{"max_local_storage_nodes":"1","master":"false"}}}}]
15/02/09 15:34:32 TRACE CommonsHttpTransport: Tx [GET]@[10.223.128.68:9500][mo-sn-2013-08/_aliases] w/ payload [null]
15/02/09 15:34:33 TRACE CommonsHttpTransport: Rx @[192.168.2.84] [200-OK] [{"mo-sn-2013-08":{"aliases":{"news":{}}}}]
15/02/09 15:34:33 WARN RestRepository: Cannot find node with id [gy_92kljRluUFT9N022KDw] (is HTTP enabled?) from shard [Shard[state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, name=16, index=mo-sn-2013-08]] in nodes [{JRpaF5f7T2uLfJ90-sLBLA=Node[id=JRpaF5f7T2uLfJ90-sLBLA, name=es-client001.nyuzu.static.ord.develop.mycomops.com, ipAddress=10.223.128.68, httpPort=9500]}]; layout [[[{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=16, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=16, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=17, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=17, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=14, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=14, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=15, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=15, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=11, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=11, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=10, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=10, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=13, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=13, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=12, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=12, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=7, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=7, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=6, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=6, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=9, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=9, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=8, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=8, index=mo-sn-2013-08}], [{state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=3, index=mo-sn-2013-08}, {state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=3, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=2, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=2, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=5, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=5, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=4, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=4, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=1, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=1, index=mo-sn-2013-08}], [{state=STARTED, primary=false, node=Q_5mAcbBS_KGm6dEVp3oqQ, relocating_node=null, shard=0, index=mo-sn-2013-08}, {state=STARTED, primary=true, node=gy_92kljRluUFT9N022KDw, relocating_node=null, shard=0, index=mo-sn-2013-08}]]]
Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cluster state volatile; cannot find node backing shards - please check whether your cluster is stable
at org.elasticsearch.hadoop.rest.RestRepository.getReadTargetShards(RestRepository.java:281)
at org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:248)
at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions$lzycompute(AbstractEsRDD.scala:49)
at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions(AbstractEsRDD.scala:48)
at org.elasticsearch.spark.rdd.AbstractEsRDD.getPartitions(AbstractEsRDD.scala:25)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:204)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:204)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:204)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1344)
at org.apache.spark.rdd.RDD.count(RDD.scala:924)
at org.apache.spark.api.java.JavaRDDLike$class.count(JavaRDDLike.scala:375)
at org.apache.spark.api.java.JavaRDD.count(JavaRDD.scala:32)
at ElasticSearchReader.main(ElasticSearchReader.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:366)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
@wingchen
Copy link
Author

Shard Config:

[ {
    "state" : "STARTED",
    "primary" : false,
    "node" : "Q_5mAcbBS_KGm6dEVp3oqQ",
    "relocating_node" : null,
    "shard" : 16,
    "index" : "mo-sn-2013-08"
  }, {
    "state" : "STARTED",
    "primary" : true,
    "node" : "gy_92kljRluUFT9N022KDw",
    "relocating_node" : null,
    "shard" : 16,
    "index" : "mo-sn-2013-08"
  } ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment