runaway ES client threads

gistfile1.txt

Suspiciously Bad Thread (one of 10 looping around in similar stack elements)

New I/O client worker #1-10 daemon (tid:26262/0x6696)
	at org.elasticsearch.common.netty.util.internal.LinkedTransferQueue.xfer(LinkedTransferQueue.java:630)
	at org.elasticsearch.common.netty.util.internal.LinkedTransferQueue.poll(LinkedTransferQueue.java:1175)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.processWriteTaskQueue(NioWorker.java:263)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:199)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)



On a normal setup when everything is quite, all ES NIO threads look like this:


New I/O client worker #1-9 daemon (tid:1132/0x46c)
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	- locked <0x00002aaabe855850> (a sun.nio.ch.Util$1)
	- locked <0x00002aaabe855838> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00002aaabe854b80> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:163)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)