Skip to content

Instantly share code, notes, and snippets.

@btmurphy-alt

btmurphy-alt/my-es-cluster.log

Last active February 21, 2019 16:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save btmurphy-alt/1564ee5b3347a9a45020a39bc2ee95a7 to your computer and use it in GitHub Desktop.
Save btmurphy-alt/1564ee5b3347a9a45020a39bc2ee95a7 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
my-es-cluster.log
[2019-02-21 07:12:46,968][INFO ][node ] [myHost] version[2.4.6], pid[10706], build[5376dca/2017-07-18T12:17:44Z]
[2019-02-21 07:12:46,968][INFO ][node ] [myHost] initializing ...
[2019-02-21 07:12:47,727][INFO ][plugins ] [myHost] modules [reindex, lang-expression, lang-groovy], plugins [license, shield], sites []
[2019-02-21 07:12:47,801][INFO ][env ] [myHost] using [1] data paths, mounts [[/ (/dev/xvda2)]], net usable_space [101.3gb], net total_space [134.3gb], spins? [no], types [ext4]
[2019-02-21 07:12:47,801][INFO ][env ] [myHost] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-02-21 07:12:48,318][INFO ][http ] [myHost] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by [shield]
[2019-02-21 07:12:48,527][INFO ][transport ] [myHost] Using [org.elasticsearch.shield.transport.ShieldServerTransportService] as transport service, overridden by [shield]
[2019-02-21 07:12:48,528][INFO ][transport ] [myHost] Using [org.elasticsearch.shield.transport.netty.ShieldNettyTransport] as transport, overridden by [shield]
[2019-02-21 07:12:49,761][DEBUG][shield.transport.netty ] [myHost] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[512kb->512kb], pipelining[true], pipelining_max_events[10000]
[2019-02-21 07:12:50,736][INFO ][node ] [myHost] initialized
[2019-02-21 07:12:50,739][INFO ][node ] [myHost] starting ...
[2019-02-21 07:12:51,084][TRACE][shield.authz.store ] [myHost] attempting to read roles file located at [/scratch/es/elasticsearch/config/shield/roles.yml]
[2019-02-21 07:12:51,136][TRACE][shield.authc.support ] [myHost] reading realm [pki/pki1] role mappings file [/scratch/es/elasticsearch/config/shield/role_mapping.yml]...
[2019-02-21 07:12:51,169][TRACE][shield.authc.esusers ] [myHost] reading users file [/scratch/es/elasticsearch/config/shield/users]...
[2019-02-21 07:12:51,174][TRACE][shield.authc.esusers ] [myHost] reading users_roles file [/scratch/es/elasticsearch/config/shield/users_roles]...
[2019-02-21 07:12:51,243][DEBUG][shield.transport.netty ] [myHost] using profile[default], worker_count[4], port[9300-9400], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb]
[2019-02-21 07:12:51,263][DEBUG][shield.transport.netty ] [myHost] binding server bootstrap to: 10.xxx.xxx.xxx
[2019-02-21 07:12:51,295][DEBUG][shield.transport.netty ] [myHost] Bound profile [default] to address {10.xxx.xxx.xxx:9300}
[2019-02-21 07:12:51,297][DEBUG][shield.transport.filter ] [myHost] loaded ip filtering profiles: [default]
[2019-02-21 07:12:51,297][INFO ][shield.transport ] [myHost] publish_address {10.xxx.xxx.xxx:9300}, bound_addresses {10.xxx.xxx.xxx:9300}
[2019-02-21 07:12:51,306][INFO ][discovery ] [myHost] kv-es-cluster/i9RZTSbOQM6jrVMNLEvWbA
[2019-02-21 07:12:51,367][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9300] to be used in ssl hostname verification
[2019-02-21 07:12:51,370][DEBUG][shield.ssl ] [myHost] using keystore[/scratch/es/elasticsearch/config/shield/elasticsearch.keys], key_algorithm[SunX509], truststore[/scratch/es/elasticsearch/config/shield/elasticsearch.trust], truststore_algorithm[PKIX], tls_protocol[TLSv1.2], session_cache_size[1000], session_cache_timeout[1d]
[2019-02-21 07:12:51,377][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9303] to be used in ssl hostname verification
[2019-02-21 07:12:51,378][TRACE][shield.transport.tracer ] [myHost] [0][internal:discovery/zen/unicast] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:51,379][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9304] to be used in ssl hostname verification
[2019-02-21 07:12:51,380][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9302] to be used in ssl hostname verification
[2019-02-21 07:12:51,380][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9300] to be used in ssl hostname verification
[2019-02-21 07:12:51,381][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9302] to be used in ssl hostname verification
[2019-02-21 07:12:51,381][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9301] to be used in ssl hostname verification
[2019-02-21 07:12:51,381][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9304] to be used in ssl hostname verification
[2019-02-21 07:12:51,382][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9301] to be used in ssl hostname verification
[2019-02-21 07:12:51,382][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9303] to be used in ssl hostname verification
[2019-02-21 07:12:51,477][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x1779b519]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,484][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xc75b97e5]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9304
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,490][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x2bdb107e]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,492][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xabc32446]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9302
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,495][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xf4a4cfcb]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9303
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,496][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x92311345]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,497][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x3c71bb27]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9302
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,498][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xb0259001]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,505][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x0290203d]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9303
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:51,509][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xd45f4dbb]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9304
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,884][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9300] to be used in ssl hostname verification
[2019-02-21 07:12:52,884][TRACE][shield.transport.tracer ] [myHost] [1][internal:discovery/zen/unicast] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:52,885][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9301] to be used in ssl hostname verification
[2019-02-21 07:12:52,885][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9301] to be used in ssl hostname verification
[2019-02-21 07:12:52,886][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9302] to be used in ssl hostname verification
[2019-02-21 07:12:52,886][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9302] to be used in ssl hostname verification
[2019-02-21 07:12:52,886][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9304] to be used in ssl hostname verification
[2019-02-21 07:12:52,887][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9303] to be used in ssl hostname verification
[2019-02-21 07:12:52,887][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9304] to be used in ssl hostname verification
[2019-02-21 07:12:52,888][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9303] to be used in ssl hostname verification
[2019-02-21 07:12:52,889][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x9f48f286]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,889][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9300] to be used in ssl hostname verification
[2019-02-21 07:12:52,894][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xd96fb7f5]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9302
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,896][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x4e814716]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,897][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x2d865b4e]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,899][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xf19d99d3]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9304
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,901][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xa48f18f0]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9302
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,906][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x5b2fccc8]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9303
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,908][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x5ae1e781]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9304
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,909][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x134f0193]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:52,911][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xf330b2cf]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9303
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,389][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9300] to be used in ssl hostname verification
[2019-02-21 07:12:54,389][TRACE][shield.transport.tracer ] [myHost] [2][internal:discovery/zen/unicast] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:54,389][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9302] to be used in ssl hostname verification
[2019-02-21 07:12:54,390][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9301] to be used in ssl hostname verification
[2019-02-21 07:12:54,390][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9304] to be used in ssl hostname verification
[2019-02-21 07:12:54,390][TRACE][shield.transport.netty ] [myHost] resolved hostname [localhost.localdomain] for address [localhost.localdomain/127.0.0.1:9303] to be used in ssl hostname verification
[2019-02-21 07:12:54,391][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9301] to be used in ssl hostname verification
[2019-02-21 07:12:54,391][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9300] to be used in ssl hostname verification
[2019-02-21 07:12:54,391][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9302] to be used in ssl hostname verification
[2019-02-21 07:12:54,392][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9304] to be used in ssl hostname verification
[2019-02-21 07:12:54,392][TRACE][shield.transport.netty ] [myHost] resolved hostname [0:0:0:0:0:0:0:1] for address [0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9303] to be used in ssl hostname verification
[2019-02-21 07:12:54,394][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x691fd57d]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,399][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xef3d3223]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9302
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,402][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x190c81b7]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9304
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,407][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x71931111]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9301
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,413][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x39b1d319]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,416][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x6b4d9c35]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9303
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,419][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x5cb7e25b]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9300
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,420][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x8990c427]]
java.net.ConnectException: Connection refused: 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:9302
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,422][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0xa0eed5de]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9304
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,426][TRACE][shield.transport.netty ] [myHost] connect exception caught on transport layer [[id: 0x8062c505]]
java.net.ConnectException: Connection refused: localhost.localdomain/127.0.0.1:9303
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-02-21 07:12:54,437][INFO ][cluster.service ] [myHost] new_master {myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2019-02-21 07:12:54,440][DEBUG][shield.authc.esnative ] [myHost] native users store waiting until gateway has recovered from disk
[2019-02-21 07:12:54,441][DEBUG][shield.authz.store ] [myHost] native roles store waiting until gateway has recovered from disk
[2019-02-21 07:12:54,441][DEBUG][shield ] [myHost] template service waiting until state has been recovered
[2019-02-21 07:12:54,511][TRACE][shield.transport.tracer ] [myHost] [3][internal:gateway/local/meta_state[n]] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:54,522][DEBUG][shield.transport.netty ] [myHost] Bound http to address {10.xxx.xxx.xxx:9200}
[2019-02-21 07:12:54,522][DEBUG][shield.transport.filter ] [myHost] loaded ip filtering profiles: [default, .http]
[2019-02-21 07:12:54,523][INFO ][http ] [myHost] publish_address {10.xxx.xxx.xxx:9200}, bound_addresses {10.xxx.xxx.xxx:9200}
[2019-02-21 07:12:54,523][INFO ][node ] [myHost] started
[2019-02-21 07:12:54,621][TRACE][shield.license ] [myHost] [shield] is running in [trial] mode
[2019-02-21 07:12:54,621][INFO ][license.plugin.core ] [myHost] license [8544165e-c6cf-43eb-96a8-a1620b6c0adb] - valid
[2019-02-21 07:12:54,628][ERROR][license.plugin.core ] [myHost]
[2019-02-21 07:12:54,643][TRACE][shield.transport.tracer ] [myHost] [4][internal:gateway/local/started_shards[n]] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:54,651][INFO ][gateway ] [myHost] recovered [1] indices into cluster_state
[2019-02-21 07:12:55,157][TRACE][shield.transport.tracer ] [myHost] [5][internal:cluster/shard/started] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:55,158][INFO ][cluster.routing.allocation] [myHost] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security][0]] ...]).
[2019-02-21 07:12:55,178][DEBUG][shield.authc.esnative ] [myHost] security index [.security] all primary shards started, so service can start
[2019-02-21 07:12:55,179][DEBUG][shield.authc.esnative ] [myHost] security index [.security] all primary shards started, so polling can start
[2019-02-21 07:12:55,179][DEBUG][shield.authz.store ] [myHost] security index [.security] all primary shards started, so polling can start
[2019-02-21 07:12:55,181][TRACE][shield.authc.esnative ] [myHost] starting polling of user index to check for changes
[2019-02-21 07:12:55,197][TRACE][shield.transport.tracer ] [myHost] [6][indices:admin/refresh[s][p]] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:55,203][TRACE][shield.action.interceptor] [myHost] intercepted request for index [.security] with neither field level or document level security not enabled, doing nothing
[2019-02-21 07:12:55,239][TRACE][shield.authz.accesscontrol] [myHost] [.security] not opting out of the query cache. request for index [.security] has field level security disabled
[2019-02-21 07:12:55,244][TRACE][shield.transport.tracer ] [myHost] [7][indices:data/read/search[phase/query+fetch]] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
[2019-02-21 07:12:55,250][TRACE][shield.transport.tracer ] [myHost] [8][indices:data/read/search[free_context/scroll]] received response from [{myHost}{i9RZTSbOQM6jrVMNLEvWbA}{myHost.us.oracle.com}{10.xxx.xxx.xxx:9300}]
*** From another command window (same host), executed the following query using PKI
--------------------------------------------
curl -k -E /scratch/es/elasticsearch-2.4.6/config/shield/elasticsearch.pem --key /scratch/es/elasticsearch-2.4.6/config/shield/elasticsearch.pkey -X GET 'https://slc00bsb:9200/_cat/plugins'
This is the response to the query
{"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/state] is unauthorized for user [myUser]"}],"type":"security_exception","reason":"action [cluster:monitor/state] is unauthorized for user [myUser]"},"status":403}
--------------------------------------------
*** This is what was written to the log file:
[2019-02-21 07:13:10,937][DEBUG][shield.ssl ] [myHost] using keystore[/scratch/es/elasticsearch/config/shield/elasticsearch.keys], key_algorithm[SunX509], truststore[/scratch/es/elasticsearch/config/shield/elasticsearch.trust], truststore_algorithm[PKIX], tls_protocol[TLSv1.2], session_cache_size[1000], session_cache_timeout[1d]
[2019-02-21 07:13:10,947][TRACE][shield.transport.netty ] [myHost] channel opened: [id: 0x7655507d, /10.xxx.xxx.xxx:29139 => /10.xxx.xxx.xxx:9200]
[2019-02-21 07:13:11,104][DEBUG][shield.authc.support ] [myHost] the roles [[]], are mapped from these [pki] groups [[]] for realm [pki/pki1]
[2019-02-21 07:13:11,111][DEBUG][shield.authc.support ] [myHost] the roles [[]], are mapped from the user [pki] for realm [CN=myUser, OU=example.org.unit, O=example.org, L=example.city, ST=example.state, C=US/pki]
*** Then I killed the ES process (ctrl-c), the final lines logged were:
[2019-02-21 07:13:11,131][TRACE][shield.transport.netty ] [myHost] channel closed: [id: 0x7655507d, /10.xxx.xxx.xxx:29139 => /10.xxx.xxx.xxx:9200]
[2019-02-21 07:13:19,959][INFO ][node ] [myHost] stopping ...
[2019-02-21 07:13:20,001][DEBUG][shield.authz.accesscontrol] [myHost] [.security] full cache clear, reason [close]
[2019-02-21 07:13:20,015][INFO ][node ] [myHost] stopped
[2019-02-21 07:13:20,016][INFO ][node ] [myHost] closing ...
[2019-02-21 07:13:20,024][INFO ][node ] [myHost] closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment