Skip to content

Instantly share code, notes, and snippets.

@takism1

takism1/es-nj01-20130805 Secret

Created August 5, 2013 21:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save takism1/0162dda53b4806d7a372 to your computer and use it in GitHub Desktop.
Save takism1/0162dda53b4806d7a372 to your computer and use it in GitHub Desktop.
Download ZIP
nj01 node log
Raw
es-nj01-20130805
[2013-08-04 06:58:47,1.2.3.4:9500]]{datacenter=ny}] transport disconnected (with verified connect)
[2013-08-04 06:58:47,1.2.3.4:9500]]{datacenter=ny}], reason [master failure, transport disconnected (with verified c
onnect)]
[2013-08-04 06:58:47,1.2.3.4:9500]]{datacenter=ny}], reason [transport disconnected (with verified connect)]
[2013-08-04 06:58:47,1.2.3.4:9500]]{datacenter=ny})]: execute
[2013-08-04 06:58:47,1.2.3.4:9500]]{datacenter=ny}], reason [possible elected master since master left (reason = t
ransport disconnected (with verified connect))]
[2013-08-04 06:58:47,1.2.3.4:
version [41], source [zen-disco-master_failed ([ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny})]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}, master
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}, local
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][X]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
---- unassigned
[2013-08-04 06:58:47,1.2.3.4:9500]]{datacenter=ny}, previous [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}}, removed {[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny},}, reason: zen-disco-master_failed ([ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny})
[2013-08-04 06:58:47,11.2.3.4:9500]]{datacenter=ny})]: done applying updated cluster_state
[2013-08-04 06:58:47,113][DEBUG][transport.netty ] [nj01] connected to node [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 06:58:47,113][DEBUG][transport.netty ] [nj01] disconnected from [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 06:59:48,129][TRACE][discovery.zen.fd ] [nj01] [master] failed to ping [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], retry [1] out of [5]
org.elasticsearch.transport.ReceiveTimeoutTransportException: [ny02][inet[/1.2.3.4:9500]][discovery/zen/fd/masterPing] request_id [314951] timed out after [60012ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-08-04 07:00:03,548][TRACE][transport.netty ] [nj01] close connection exception caught on transport layer [[id: 0x371c0372, /1.2.3.4:36059 => /1.2.3.4:9500]], disconnecting from relevant node
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:1.2.3.4:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:1.2.3.4:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-08-04 07:00:03,549][TRACE][transport.netty ] [nj01] close connection exception caught on transport layer [[id: 0xb534da71, /1.2.3.4:36066 => /1.2.3.4:9500]], disconnecting from relevant node
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:1.2.3.4:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:1.2.3.4:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-08-04 07:00:03,549][TRACE][transport.netty ] [nj01] channel closed: [id: 0xb534da71, /1.2.3.4:36066 => /1.2.3.4:9500]
[2013-08-04 07:00:03,548][TRACE][transport.netty ] [nj01] close connection exception caught on transport layer [[id: 0x185d74a9, /1.2.3.4:36058 => /1.2.3.4:9500]], disconnecting from relevant node
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:1.2.3.4:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:1.2.3.4:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-08-04 07:00:03,549][TRACE][transport.netty ] [nj01] channel closed: [id: 0x371c0372, /1.2.3.4:36059 => /1.2.3.4:9500]
[2013-08-04 07:00:03,552][TRACE][transport.netty ] [nj01] channel closed: [id: 0x185d74a9, /1.2.3.4:36058 => /1.2.3.4:9500]
[2013-08-04 07:00:03,693][TRACE][transport.netty ] [nj01] channel closed: [id: 0xed282965, /1.2.3.4:36063 => /1.2.3.4:9500]
[2013-08-04 07:00:03,693][TRACE][transport.netty ] [nj01] channel closed: [id: 0x68fc9ddc, /1.2.3.4:36065 => /1.2.3.4:9500]
[2013-08-04 07:00:03,693][TRACE][transport.netty ] [nj01] channel closed: [id: 0xa6a27cc9, /1.2.3.4:36060 => /1.2.3.4:9500]
[2013-08-04 07:00:03,693][TRACE][transport.netty ] [nj01] channel closed: [id: 0xe0ca6b3a, /1.2.3.4:36064 => /1.2.3.4:9500]
[2013-08-04 07:00:03,693][TRACE][transport.netty ] [nj01] channel closed: [id: 0x8bd9128a, /1.2.3.4:36061 => /1.2.3.4:9500]
[2013-08-04 07:00:03,693][TRACE][transport.netty ] [nj01] channel closed: [id: 0x423b1692, /1.2.3.4:36062 => /1.2.3.4:9500]
[2013-08-04 07:00:05,309][TRACE][transport.netty ] [nj01] close connection exception caught on transport layer [[id: 0x6af5def6, /1.2.3.4:40339 => /1.2.3.4:9500]], disconnecting from relevant node
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:1.2.3.4:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:1.2.3.4:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[2013-08-04 07:00:05,309][DEBUG][transport.netty ] [nj01] disconnected from [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], channel closed event
[2013-08-04 07:00:05,315][DEBUG][transport.netty ] [nj01] connected to node [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:06,339][DEBUG][discovery.zen.fd ] [nj01] [master] pinging a master [ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny} that is no longer a master
[2013-08-04 07:00:06,340][INFO ][discovery.zen ] [nj01] master_left [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], reason [no longer master]
[2013-08-04 07:00:06,340][DEBUG][cluster.service ] [nj01] processing [zen-disco-master_failed ([ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny})]: execute
[2013-08-04 07:00:06,340][DEBUG][discovery.zen.fd ] [nj01] [master] stopping fault detection against master [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], reason [master failure, no longer master]
[2013-08-04 07:00:06,340][WARN ][discovery.zen ] [nj01] not enough master nodes after master left (reason = no longer master), current nodes: {[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj},[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj},}
[2013-08-04 07:00:06,340][TRACE][cluster.service ] [nj01] cluster state updated:
version [41], source [zen-disco-master_failed ([ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny})]
nodes:
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}, local
routing_table:
routing_nodes:
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
---- unassigned
[2013-08-04 07:00:06,340][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[2013-08-04 07:00:06,340][INFO ][cluster.service ] [nj01] removed {[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj},[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny},}, reason: zen-disco-master_failed ([ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny})
[2013-08-04 07:00:06,341][TRACE][discovery.zen.ping.unicast] [nj01] [5] connecting (light) to [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]
[2013-08-04 07:00:06,341][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}]
[2013-08-04 07:00:06,342][TRACE][discovery.zen.ping.unicast] [nj01] [5] connecting (light) to [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]
[2013-08-04 07:00:06,343][TRACE][discovery.zen.ping.unicast] [nj01] [5] connecting (light) to [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]
[2013-08-04 07:00:06,347][DEBUG][transport.netty ] [nj01] connected to node [[#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]]
[2013-08-04 07:00:06,347][TRACE][discovery.zen.ping.unicast] [nj01] [5] connected to [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]
[2013-08-04 07:00:06,347][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]
[2013-08-04 07:00:06,347][DEBUG][cluster.service ] [nj01] processing [zen-disco-master_failed ([ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny})]: done applying updated cluster_state
[2013-08-04 07:00:06,348][DEBUG][transport.netty ] [nj01] disconnected from [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}]
[2013-08-04 07:00:06,348][DEBUG][transport.netty ] [nj01] disconnected from [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:06,349][DEBUG][transport.netty ] [nj01] connected to node [[#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]]
[2013-08-04 07:00:06,349][TRACE][discovery.zen.ping.unicast] [nj01] [5] connected to [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]
[2013-08-04 07:00:06,349][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]
[2013-08-04 07:00:06,349][DEBUG][transport.netty ] [nj01] connected to node [[#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]]
[2013-08-04 07:00:06,349][TRACE][discovery.zen.ping.unicast] [nj01] [5] connected to [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]
[2013-08-04 07:00:06,349][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]
[2013-08-04 07:00:06,350][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:06,351][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:06,352][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:07,848][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[2013-08-04 07:00:07,849][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]
[2013-08-04 07:00:07,849][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]
[2013-08-04 07:00:07,849][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]
[2013-08-04 07:00:07,849][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}]
[2013-08-04 07:00:07,850][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:07,852][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:07,852][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:09,368][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}
[2013-08-04 07:00:09,369][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]
[2013-08-04 07:00:09,369][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]
[2013-08-04 07:00:09,369][TRACE][discovery.zen.ping.unicast] [nj01] [5] sending to [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]
[2013-08-04 07:00:09,369][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}]
[2013-08-04 07:00:09,370][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:09,371][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:09,372][TRACE][discovery.zen.ping.unicast] [nj01] [5] received response from [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]: [ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}], master [null], cluster_name[newspr_production]}, ping_response{target [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], cluster_name[newspr_production]}]
[2013-08-04 07:00:09,372][TRACE][discovery.zen.ping.unicast] [nj01] [5] disconnecting from [#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]
[2013-08-04 07:00:09,372][DEBUG][transport.netty ] [nj01] disconnected from [[#zen_unicast_4#][inet[ny02/1.2.3.4:9500]]]
[2013-08-04 07:00:09,372][TRACE][discovery.zen.ping.unicast] [nj01] [5] disconnecting from [#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]
[2013-08-04 07:00:09,372][DEBUG][transport.netty ] [nj01] disconnected from [[#zen_unicast_3#][inet[ny01/1.2.3.4:9500]]]
[2013-08-04 07:00:09,372][TRACE][discovery.zen.ping.unicast] [nj01] [5] disconnecting from [#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]
[2013-08-04 07:00:09,372][DEBUG][transport.netty ] [nj01] disconnected from [[#zen_unicast_2#][inet[nj02/1.2.3.4:9500]]]
[2013-08-04 07:00:09,373][TRACE][discovery.zen ] [nj01] full ping responses:
--> target [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
--> target [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
--> target [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,373][DEBUG][discovery.zen ] [nj01] filtered ping responses: (filter_client[true], filter_data[false])
--> target [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
--> target [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
--> target [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,379][DEBUG][transport.netty ] [nj01] connected to node [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,385][DEBUG][discovery.zen.fd ] [nj01] [master] starting fault detection against master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}], reason [initial_join]
[2013-08-04 07:00:09,386][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: execute
[2013-08-04 07:00:09,387][DEBUG][cluster.service ] [nj01] got first state from fresh master [HPRnuckbQg2-r-Hf9jBXcg]
[2013-08-04 07:00:09,387][TRACE][cluster.service ] [nj01] cluster state updated:
version [42], source [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, master
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}, local
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
---- unassigned
[2013-08-04 07:00:09,387][INFO ][cluster.service ] [nj01] detected_master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, added {[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj},[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny},[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny},}, reason: zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])
[2013-08-04 07:00:09,389][DEBUG][transport.netty ] [nj01] connected to node [[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}]
[2013-08-04 07:00:09,394][DEBUG][transport.netty ] [nj01] connected to node [[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}]
[2013-08-04 07:00:09,437][WARN ][indices.cluster ] [nj01] [my_index][0] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,437][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,438][WARN ][indices.cluster ] [nj01] [my_index][1] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,438][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,438][WARN ][indices.cluster ] [nj01] [my_index][2] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,438][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,438][WARN ][indices.cluster ] [nj01] [my_index][3] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,438][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,439][TRACE][gateway.local.state.meta ] [nj01] [_global] writing state, reason [changed]
[2013-08-04 07:00:09,441][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: done applying updated cluster_state
[2013-08-04 07:00:09,446][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: execute
[2013-08-04 07:00:09,447][TRACE][cluster.service ] [nj01] cluster state updated:
version [43], source [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, master
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}, local
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][0], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
---- unassigned
--------[my_index][0], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,483][WARN ][indices.cluster ] [nj01] [my_index][1] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,484][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,484][WARN ][indices.cluster ] [nj01] [my_index][2] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,484][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,484][WARN ][indices.cluster ] [nj01] [my_index][3] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,484][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,485][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: done applying updated cluster_state
[2013-08-04 07:00:09,485][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: execute
[2013-08-04 07:00:09,485][TRACE][cluster.service ] [nj01] cluster state updated:
version [44], source [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, master
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}, local
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[null], [R], s[UNASSIGNED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
---- unassigned
--------[my_index][3], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,551][WARN ][indices.cluster ] [nj01] [my_index][1] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,551][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][1], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,551][WARN ][indices.cluster ] [nj01] [my_index][2] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,551][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,551][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: done applying updated cluster_state
[2013-08-04 07:00:09,551][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: execute
[2013-08-04 07:00:09,552][TRACE][cluster.service ] [nj01] cluster state updated:
version [45], source [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, master
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}, local
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
---- unassigned
--------[my_index][1], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,552][WARN ][indices.cluster ] [nj01] [my_index][2] master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}] marked shard as started, but shard has not been created, mark shard as failed
[2013-08-04 07:00:09,552][WARN ][cluster.action.shard ] [nj01] sending failed shard for [my_index][2], node[fy3VCsuOSm-eqI35T4afSg], [R], s[STARTED], reason [master [ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny} marked shard as started, but shard has not been created, mark shard as failed]
[2013-08-04 07:00:09,581][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: done applying updated cluster_state
[2013-08-04 07:00:09,582][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: execute
[2013-08-04 07:00:09,582][TRACE][cluster.service ] [nj01] cluster state updated:
version [46], source [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]
nodes:
[nj02][mhhvyRRoQ6WQX52Ov-GkjQ][inet[/1.2.3.4:9500]]{datacenter=nj}
[ny02][NRPvTu1yToiC7SKOxPxNeA][inet[/1.2.3.4:9500]]{datacenter=ny}
[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}, master
[nj01][fy3VCsuOSm-eqI35T4afSg][inet[/1.2.3.4:9500]]{datacenter=nj}, local
routing_table:
-- index [my_index]
----shard_id [my_index][0]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
----shard_id [my_index][1]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][2]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[null], [R], s[UNASSIGNED]
----shard_id [my_index][3]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
routing_nodes:
-----node_id[mhhvyRRoQ6WQX52Ov-GkjQ][V]
--------[my_index][0], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][1], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][2], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
--------[my_index][3], node[mhhvyRRoQ6WQX52Ov-GkjQ], [R], s[STARTED]
-----node_id[NRPvTu1yToiC7SKOxPxNeA][V]
--------[my_index][0], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][1], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][2], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
--------[my_index][3], node[NRPvTu1yToiC7SKOxPxNeA], [P], s[STARTED]
-----node_id[HPRnuckbQg2-r-Hf9jBXcg][V]
--------[my_index][0], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][1], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][2], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
--------[my_index][3], node[HPRnuckbQg2-r-Hf9jBXcg], [R], s[STARTED]
-----node_id[fy3VCsuOSm-eqI35T4afSg][V]
--------[my_index][0], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
--------[my_index][3], node[fy3VCsuOSm-eqI35T4afSg], [R], s[INITIALIZING]
---- unassigned
--------[my_index][1], node[null], [R], s[UNASSIGNED]
--------[my_index][2], node[null], [R], s[UNASSIGNED]
[2013-08-04 07:00:09,583][DEBUG][cluster.service ] [nj01] processing [zen-disco-receive(from master [[ny01][HPRnuckbQg2-r-Hf9jBXcg][inet[/1.2.3.4:9500]]{datacenter=ny}])]: done applying updated cluster_state
[2013-08-04 07:00:11,142][TRACE][transport.netty ] [nj01] channel opened: [id: 0x9af672e5, /1.2.3.4:36625 => /1.2.3.4:9500]
[2013-08-04 07:00:11,142][TRACE][transport.netty ] [nj01] channel opened: [id: 0x2d9f5929, /1.2.3.4:36626 => /1.2.3.4:9500]
[2013-08-04 07:00:11,142][TRACE][transport.netty ] [nj01] channel opened: [id: 0x1209b4af, /1.2.3.4:36627 => /1.2.3.4:9500]
[2013-08-04 07:00:11,143][TRACE][transport.netty ] [nj01] channel opened: [id: 0x30378382, /1.2.3.4:36628 => /1.2.3.4:9500]
[2013-08-04 07:00:11,143][TRACE][transport.netty ] [nj01] channel opened: [id: 0xe2d3b712, /1.2.3.4:36629 => /1.2.3.4:9500]
[2013-08-04 07:00:11,143][TRACE][transport.netty ] [nj01] channel opened: [id: 0xb52cee60, /1.2.3.4:36630 => /1.2.3.4:9500]
[2013-08-04 07:00:11,143][TRACE][transport.netty ] [nj01] channel opened: [id: 0x49f18f1c, /1.2.3.4:36631 => /1.2.3.4:9500]
[2013-08-04 07:00:11,143][TRACE][transport.netty ] [nj01] channel opened: [id: 0x797e1b54, /1.2.3.4:36632 => /1.2.3.4:9500]
[2013-08-04 07:00:11,144][TRACE][transport.netty ] [nj01] channel opened: [id: 0xc7164c06, /1.2.3.4:36633 => /1.2.3.4:9500]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment