Skip to content

Instantly share code, notes, and snippets.

@ollez
Last active December 21, 2015 17:38
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save ollez/d0450930a591912aea5d to your computer and use it in GitHub Desktop.
postPut clj code
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.46.9.117:63731 remote=10.46.9.117/10.46.9.117:60020]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:373)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580)
13/08/26 15:51:13 WARN client.HConnectionManager$HConnectionImplementation: Failed all from region=sessionpartials,,1377524862165.0eeea159235b200e09127f04bedfaa5b., hostname=10.46.9.117, port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Call to 10.46.9.117/10.46.9.117:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.46.9.117:63731 remote=10.46.9.117/10.46.9.117:60020]
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1710)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:800)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:395)
at com.digitalroute.omniscience.SessionService$2.onNext(SessionService.java:99)
at com.digitalroute.omniscience.SessionService$2.onNext(SessionService.java:1)
at rx.operators.AtomicObserver.onNext(AtomicObserver.java:90)
at rx.operators.OperationTakeLast$TakeLast$ItemObserver.onCompleted(OperationTakeLast.java:75)
at rx.operators.OperationScan$AccumulatingObserver.onCompleted(OperationScan.java:158)
at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omniscience.serialization$eval445$fn__446$fn__447$fn__448.invoke(Unknown Source)
at clojure.core$binding_conveyor_fn$fn__4107.invoke(core.clj:1836)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketTimeoutException: Call to 10.46.9.117/10.46.9.117:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.46.9.117:63731 remote=10.46.9.117/10.46.9.117:60020]
at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1029)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1002)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at $Proxy25.multi(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1539)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3$1.call(HConnectionManager.java:1537)
at org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.java:215)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1546)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$3.call(HConnectionManager.java:1534)
... 5 more
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.46.9.117:63731 remote=10.46.9.117/10.46.9.117:60020]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:373)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580)
13/08/26 15:51:13 WARN client.HConnectionManager$HConnectionImplementation: Failed all from region=sessionpartials,,1377524862165.0eeea159235b200e09127f04bedfaa5b., hostname=10.46.9.117, port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Call to 10.46.9.117/10.46.9.117:60020 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.46.9.117:63731 remote=10.46.9.117/10.46.9.117:60020]
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
at java.util.concurrent.FutureTask.get(FutureTask.java:111)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1710)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:800)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:395)
at com.digitalroute.omniscience.SessionService$2.onNext(SessionService.java:99)
at com.digitalroute.omniscience.SessionService$2.onNext(SessionService.java:1)
at rx.operators.AtomicObserver.onNext(AtomicObserver.java:90)
at rx.operators.OperationTakeLast$TakeLast$ItemObserver.onCompleted(OperationTakeLast.java:75)
at rx.operators.OperationScan$AccumulatingObserver.onCompleted(OperationScan.java:158)
at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omniscience.serialization$eval445$fn__446$fn__447$fn__448.invoke(Unknown Source)
at clojure.core$binding_conveyor_fn$fn__4107.invoke(core.clj:1836)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
(defn -postPut
[this ctx put wal-edit write-to-wal?]
(if (= (.getBytes "sessionpartials") (.getRow put))
((get-logger this) (str "########### POST PUT #############" put))
(let [log (get-logger this)
query (get-query this)
env (.getEnvironment ctx)
region (.getRegion env)
row-key (.getRow put)
get (Get. row-key)
encoded-vals (.values (.getFamilyMap (.get region get) (string-to-bytes "pcf")))
decoded-vals (map taoensso.nippy/thaw-from-bytes encoded-vals)
session-aggregates-table (.getTable env (string-to-bytes "sessionaggregates"))
put (Put. row-key)
rs (pr-str (q query decoded-vals))
rb (str->gzipped-bytes rs)]
(try
(.setAutoFlush session-aggregates-table false)
(.add put (string-to-bytes "scf")
(string-to-bytes "val")
rb)
(.put session-aggregates-table put)
(finally
(.flushCommits session-aggregates-table) ;; <----- LOTS OF THREADS HANG HERE
(.close session-aggregates-table)
(log (str "FLUSHED " put)))))))
<configuration>
<property>
<name>hbase.regionserver.handler.count</name>
<value>10</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>
<property>
<name>hbase.regionserver.lease.period</name>
<value>900000</value> <!-- 15 minutes -->
</property>
<property>
<name>hbase.rpc.timeout</name>
<value>900000></value> <!-- 15 minutes -->
</property>
</configuration>
2013-08-26 16:12:05
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.3-b01 mixed mode):
"IPC Client (47) connection to localhost/127.0.0.1:9000 from ollem" daemon prio=5 tid=0x00007febec304000 nid=0x13203 in Object.wait() [0x0000000117ae9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000070891a490> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
- locked <0x000000070891a490> (a org.apache.hadoop.ipc.Client$Connection)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)
"IPC Client (47) connection to 10.46.9.117/10.46.9.117:60020 from ollem" daemon prio=5 tid=0x00007febe8cc3000 nid=0x13103 runnable [0x0000000117540000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007222905b8> (a sun.nio.ch.Util$2)
- locked <0x00000007222905c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000722290568> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:373)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x0000000702eb7448> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580)
"Attach Listener" daemon prio=5 tid=0x00007febeca0d800 nid=0x13007 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"RS_CLOSE_REGION-10.46.9.117,60020,1377524450250-2" prio=5 tid=0x00007febe8aee800 nid=0xdf03 waiting on condition [0x0000000117746000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223a24a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"RS_CLOSE_REGION-10.46.9.117,60020,1377524450250-1" prio=5 tid=0x00007febe9842000 nid=0xda03 waiting on condition [0x000000011743d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223a24a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"RS_CLOSE_REGION-10.46.9.117,60020,1377524450250-0" prio=5 tid=0x00007febec20e800 nid=0xd503 waiting on condition [0x0000000116e2b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223a24a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"ResponseProcessor for block blk_7966217991170375192_2507" daemon prio=5 tid=0x00007febea806000 nid=0xd403 runnable [0x0000000117643000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007223a2698> (a sun.nio.ch.Util$2)
- locked <0x00000007223a26a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007223a2648> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2967)
"DataStreamer for file /hbase/.logs/10.46.9.117,60020,1377524450250/10.46.9.117%2C60020%2C1377524450250.1377524862247 block blk_7966217991170375192_2507" daemon prio=5 tid=0x00007febec136000 nid=0xcf03 in Object.wait() [0x0000000117237000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007222929f0> (a java.util.LinkedList)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2806)
- locked <0x00000007222929f0> (a java.util.LinkedList)
"RS_OPEN_REGION-10.46.9.117,60020,1377524450250-2" prio=5 tid=0x00007febec881800 nid=0xb303 waiting on condition [0x000000011733a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722347178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"RS_OPEN_REGION-10.46.9.117,60020,1377524450250-1" prio=5 tid=0x00007febea805800 nid=0xb003 waiting on condition [0x0000000117134000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722347178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"RS_OPEN_REGION-10.46.9.117,60020,1377524450250-0" prio=5 tid=0x00007febe8c20000 nid=0xae03 waiting on condition [0x00000001144fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722347178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"PostOpenDeployTasks:1028785192-EventThread" daemon prio=5 tid=0x00007febe8c16800 nid=0xa903 waiting on condition [0x0000000117031000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722292be0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"PostOpenDeployTasks:1028785192-SendThread(fe80:0:0:0:0:0:0:1%1:2181)" daemon prio=5 tid=0x00007febe8aff800 nid=0xa803 runnable [0x0000000116f2e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007223472e0> (a sun.nio.ch.Util$2)
- locked <0x00000007223472f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000722347290> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"RS_OPEN_META-10.46.9.117,60020,1377524450250-0" prio=5 tid=0x00007febe9cb3000 nid=0xa203 waiting on condition [0x0000000114011000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722292cd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"RS_OPEN_ROOT-10.46.9.117,60020,1377524450250-0" prio=5 tid=0x00007febe996d800 nid=0x9c03 waiting on condition [0x00000001118a5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fdbb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"SplitLogWorker-10.46.9.117,60020,1377524450250" prio=5 tid=0x00007febe8bcd000 nid=0x9303 in Object.wait() [0x0000000116c27000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722292e40> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:219)
- locked <0x0000000722292e40> (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
at java.lang.Thread.run(Thread.java:722)
"REPL IPC Server handler 2 on 60020" daemon prio=5 tid=0x00007febed047000 nid=0x9203 waiting on condition [0x0000000116b24000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fdd30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"REPL IPC Server handler 1 on 60020" daemon prio=5 tid=0x00007febe9a13800 nid=0x9103 waiting on condition [0x0000000116a21000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fdd30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"REPL IPC Server handler 0 on 60020" daemon prio=5 tid=0x00007febec212800 nid=0x9003 waiting on condition [0x000000011691e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fdd30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 9 on 60020" daemon prio=5 tid=0x00007febea832000 nid=0x8f03 waiting on condition [0x000000011681b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 8 on 60020" daemon prio=5 tid=0x00007febea831800 nid=0x8e03 waiting on condition [0x0000000116718000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 7 on 60020" daemon prio=5 tid=0x00007febea82e800 nid=0x8d03 waiting on condition [0x0000000116615000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 6 on 60020" daemon prio=5 tid=0x00007febe8bcc800 nid=0x8c03 waiting on condition [0x00000001153ae000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 5 on 60020" daemon prio=5 tid=0x00007febed044000 nid=0x8b03 waiting on condition [0x0000000111b40000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 4 on 60020" daemon prio=5 tid=0x00007febed043000 nid=0x8a03 waiting on condition [0x0000000116512000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 3 on 60020" daemon prio=5 tid=0x00007febe8c81000 nid=0x8903 waiting on condition [0x000000011640f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 2 on 60020" daemon prio=5 tid=0x00007febe8d73000 nid=0x8803 waiting on condition [0x000000011630c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 1 on 60020" daemon prio=5 tid=0x00007febeb048000 nid=0x8703 waiting on condition [0x0000000116209000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"PRI IPC Server handler 0 on 60020" daemon prio=5 tid=0x00007febed042800 nid=0x8603 waiting on condition [0x0000000116106000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007223fde30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1398)
"IPC Server handler 9 on 60020" daemon prio=5 tid=0x00007febec1ce800 nid=0x8503 waiting for monitor entry [0x0000000116002000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 8 on 60020" daemon prio=5 tid=0x00007febed041800 nid=0x8403 waiting for monitor entry [0x0000000115eff000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 7 on 60020" daemon prio=5 tid=0x00007febed041000 nid=0x8303 waiting for monitor entry [0x0000000115dfc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 6 on 60020" daemon prio=5 tid=0x00007febed040000 nid=0x8203 waiting for monitor entry [0x0000000115cf9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 5 on 60020" daemon prio=5 tid=0x00007febed07d800 nid=0x8103 waiting for monitor entry [0x0000000115bf6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 4 on 60020" daemon prio=5 tid=0x00007febed07c800 nid=0x8003 waiting for monitor entry [0x0000000115af3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 3 on 60020" daemon prio=5 tid=0x00007febed07c000 nid=0x7f03 waiting for monitor entry [0x00000001159f0000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 2 on 60020" daemon prio=5 tid=0x00007febe8e9f000 nid=0x7e03 waiting for monitor entry [0x00000001158ed000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 1 on 60020" daemon prio=5 tid=0x00007febe8cc9000 nid=0x7d03 waiting for monitor entry [0x00000001157ea000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1115)
- waiting to lock <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server handler 0 on 60020" daemon prio=5 tid=0x00007febe8c08800 nid=0x7c03 in Object.wait() [0x00000001152a9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000702eb67f8> (a org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:984)
- locked <0x0000000702eb67f8> (a org.apache.hadoop.hbase.ipc.HBaseClient$Call)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at $Proxy13.getClosestRowBefore(Unknown Source)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1137)
- locked <0x0000000722d116e8> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1000)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:975)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:875)
at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:83)
at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:163)
at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:658)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:148)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:1062)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1124)
- locked <0x00000007223fe110> (a java.lang.Object)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1677)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1562)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:966)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment$HTableWrapper.flushCommits(CoprocessorHost.java:446)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
at omnicp.put_cp$_postPut.invoke(put_cp.clj:129)
at omnicp.Put.postPut(Unknown Source)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postPut(RegionCoprocessorHost.java:834)
at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2504)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2174)
at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
"IPC Server listener on 60020" daemon prio=5 tid=0x00007febed037000 nid=0x7b03 waiting for monitor entry [0x00000001156e8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.registerChannel(HBaseServer.java:569)
- waiting to lock <0x000000072234ffa0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doAccept(HBaseServer.java:721)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:644)
"IPC Server Responder" daemon prio=5 tid=0x00007febed036000 nid=0x7a03 runnable [0x00000001155e5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x0000000722348970> (a sun.nio.ch.Util$2)
- locked <0x0000000722348960> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007223488c0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRunLoop(HBaseServer.java:825)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:808)
"Timer-0" daemon prio=5 tid=0x00007febe8bda000 nid=0x7903 in Object.wait() [0x00000001151a8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722459870> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000722459870> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"285806753@qtp-1439108780-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60030" prio=5 tid=0x00007febec811000 nid=0x7803 runnable [0x000000011501e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007224599a8> (a sun.nio.ch.Util$2)
- locked <0x00000007224599b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000722459958> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
"1185626597@qtp-1439108780-0" prio=5 tid=0x00007febec806000 nid=0x7703 in Object.wait() [0x0000000114f1b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722348bd0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
- locked <0x0000000722348bd0> (a org.mortbay.thread.QueuedThreadPool$PoolThread)
"regionserver60020.leaseChecker" daemon prio=5 tid=0x00007febe8d75000 nid=0x7603 waiting on condition [0x0000000114e18000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007222ec640> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:256)
at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:83)
at java.lang.Thread.run(Thread.java:722)
"regionserver60020.periodicFlusher" daemon prio=5 tid=0x00007febe8d74000 nid=0x7503 in Object.wait() [0x0000000114d15000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722459c40> (a java.lang.Object)
at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
- locked <0x0000000722459c40> (a java.lang.Object)
at org.apache.hadoop.hbase.Chore.run(Chore.java:75)
at java.lang.Thread.run(Thread.java:722)
"regionserver60020.compactionChecker" daemon prio=5 tid=0x00007febe8c1a800 nid=0x7403 in Object.wait() [0x0000000114c12000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722459d00> (a java.lang.Object)
at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
- locked <0x0000000722459d00> (a java.lang.Object)
at org.apache.hadoop.hbase.Chore.run(Chore.java:75)
at java.lang.Thread.run(Thread.java:722)
"regionserver60020.cacheFlusher" daemon prio=5 tid=0x00007febe8938000 nid=0x7303 waiting on condition [0x0000000114b0f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722459dc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:256)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)
at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:220)
at java.lang.Thread.run(Thread.java:722)
"regionserver60020.logRoller" daemon prio=5 tid=0x00007febe8e2c000 nid=0x7203 in Object.wait() [0x0000000114a0c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007224b4d28> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:77)
- locked <0x00000007224b4d28> (a java.util.concurrent.atomic.AtomicBoolean)
at java.lang.Thread.run(Thread.java:722)
"Timer thread for monitoring jvm" daemon prio=5 tid=0x00007febec817800 nid=0x7103 in Object.wait() [0x0000000114909000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007224b4de0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007224b4de0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Timer thread for monitoring hbase" daemon prio=5 tid=0x00007febec812800 nid=0x7003 in Object.wait() [0x0000000114806000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007224b4ea0> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007224b4ea0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"regionserver60020.logSyncer" daemon prio=5 tid=0x00007febec807800 nid=0x6f03 in Object.wait() [0x0000000114703000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722510840> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1237)
- locked <0x0000000722510840> (a java.util.concurrent.atomic.AtomicBoolean)
at java.lang.Thread.run(Thread.java:722)
"LeaseChecker" daemon prio=5 tid=0x00007febe8810800 nid=0x4b07 waiting on condition [0x0000000114600000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
at java.lang.Thread.run(Thread.java:722)
"IPC Client (47) connection to /10.46.9.117:60000 from ollem" daemon prio=5 tid=0x00007febe9829800 nid=0x3f07 in Object.wait() [0x00000001142f7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007224b4f78> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:534)
- locked <0x00000007224b4f78> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:579)
"regionserver60020-EventThread" daemon prio=5 tid=0x00007febe9804800 nid=0x6e03 waiting on condition [0x0000000113b42000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007225108f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
"regionserver60020-SendThread(fe80:0:0:0:0:0:0:1%1:2181)" daemon prio=5 tid=0x00007febe98f2000 nid=0x6d03 runnable [0x0000000111c52000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000000072234c560> (a sun.nio.ch.Util$2)
- locked <0x000000072234c550> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000072234c2c8> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
"DestroyJavaVM" prio=5 tid=0x00007febea008000 nid=0x1b03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"regionserver60020" prio=5 tid=0x00007febe8b6b800 nid=0x6c03 in Object.wait() [0x0000000113f0e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007224b50d8> (a java.lang.Object)
at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
- locked <0x00000007224b50d8> (a java.lang.Object)
at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:55)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:795)
at java.lang.Thread.run(Thread.java:722)
"LRU Statistics #0" daemon prio=5 tid=0x00007febe8b6b000 nid=0x6b03 waiting on condition [0x0000000113e0b000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007225109e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"main.LruBlockCache.EvictionThread" daemon prio=5 tid=0x00007febec0f6800 nid=0x6a07 in Object.wait() [0x0000000113d08000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722510f28> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:631)
- locked <0x0000000722510f28> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
at java.lang.Thread.run(Thread.java:722)
"Timer thread for monitoring rpc" daemon prio=5 tid=0x00007febe8ba7000 nid=0x6703 in Object.wait() [0x000000011281a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007222ec758> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007222ec758> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"IPC Reader 9 on port 60020" daemon prio=5 tid=0x00007febe8b54800 nid=0x6603 waiting on condition [0x0000000112717000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722459f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1335)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
- locked <0x000000072234fb98> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 8 on port 60020" daemon prio=5 tid=0x00007febe8b54000 nid=0x6503 waiting on condition [0x0000000112614000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722459f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1335)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
- locked <0x000000072234ff88> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 7 on port 60020" daemon prio=5 tid=0x00007febe8b1b800 nid=0x6403 runnable [0x0000000112511000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000000072234e848> (a sun.nio.ch.Util$2)
- locked <0x000000072234e838> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000072234e7a8> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
- locked <0x000000072234ece0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 6 on port 60020" daemon prio=5 tid=0x00007febe8c0f000 nid=0x6303 runnable [0x000000011240e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000000072234eed0> (a sun.nio.ch.Util$2)
- locked <0x000000072234eec0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000072234ee30> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
- locked <0x000000072234f2d8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 5 on port 60020" daemon prio=5 tid=0x00007febe8bfd800 nid=0x6203 runnable [0x000000011230b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000007224b51e8> (a sun.nio.ch.Util$2)
- locked <0x00000007224b51f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007224b5198> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
- locked <0x00000007224b5208> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 4 on port 60020" daemon prio=5 tid=0x00007febe8d64800 nid=0x6103 waiting on condition [0x0000000112208000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722459f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1335)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
- locked <0x000000072234ffa0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 3 on port 60020" daemon prio=5 tid=0x00007febe8acd000 nid=0x6003 waiting on condition [0x0000000112105000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722459f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1335)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
- locked <0x000000072234f8d0> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 2 on port 60020" daemon prio=5 tid=0x00007febe8c7e000 nid=0x5f03 runnable [0x0000000112002000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:159)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x000000072245b960> (a sun.nio.ch.Util$2)
- locked <0x000000072245b950> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000072245b8c0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:528)
- locked <0x0000000722406a50> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 1 on port 60020" daemon prio=5 tid=0x00007febe8d5b000 nid=0x5e03 waiting on condition [0x0000000111eff000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722459f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1335)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
- locked <0x0000000722406a68> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Reader 0 on port 60020" daemon prio=5 tid=0x00007febe8c22800 nid=0x5a07 waiting on condition [0x0000000111dfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000722459f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1335)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
- locked <0x000000072234f8e8> (a org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"Service Thread" daemon prio=5 tid=0x00007febe893a000 nid=0x5703 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=5 tid=0x00007febe8939000 nid=0x5603 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=5 tid=0x00007febe8937800 nid=0x5503 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=5 tid=0x00007febeb000000 nid=0x5403 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=0x00007febec015000 nid=0x5213 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=5 tid=0x00007febe8936800 nid=0x3e03 in Object.wait() [0x000000011095d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000722406e30> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x0000000722406e30> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=5 tid=0x00007febe8936000 nid=0x3d03 in Object.wait() [0x000000011085a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007222ec630> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000007222ec630> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=5 tid=0x00007febe8933000 nid=0x3c03 runnable
"Gang worker#0 (Parallel GC Threads)" prio=5 tid=0x00007febe8811000 nid=0x3103 runnable
"Gang worker#1 (Parallel GC Threads)" prio=5 tid=0x00007febe881d800 nid=0x3203 runnable
"Gang worker#2 (Parallel GC Threads)" prio=5 tid=0x00007febe881e000 nid=0x3303 runnable
"Gang worker#3 (Parallel GC Threads)" prio=5 tid=0x00007febe881e800 nid=0x3403 runnable
"Gang worker#4 (Parallel GC Threads)" prio=5 tid=0x00007febe881f000 nid=0x3503 runnable
"Gang worker#5 (Parallel GC Threads)" prio=5 tid=0x00007febe8820000 nid=0x3603 runnable
"Gang worker#6 (Parallel GC Threads)" prio=5 tid=0x00007febe8820800 nid=0x3703 runnable
"Gang worker#7 (Parallel GC Threads)" prio=5 tid=0x00007febe9805800 nid=0x3803 runnable
"Concurrent Mark-Sweep GC Thread" prio=5 tid=0x00007febe88f1000 nid=0x3b03 runnable
"Gang worker#0 (Parallel CMS Threads)" prio=5 tid=0x00007febe88f0000 nid=0x3903 runnable
"Gang worker#1 (Parallel CMS Threads)" prio=5 tid=0x00007febe88f0800 nid=0x3a03 runnable
"VM Periodic Task Thread" prio=5 tid=0x00007febe893b800 nid=0x5803 waiting on condition
JNI global references: 170
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment