Skip to content

Instantly share code, notes, and snippets.

@torao
Created July 3, 2012 02:58
Show Gist options
  • Save torao/3037247 to your computer and use it in GitHub Desktop.
Save torao/3037247 to your computer and use it in GitHub Desktop.
StackTrace of TaskTracker in Load Spiking after Jun 30 2012 Leapsecond
2012-07-02 19:30:36
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.0-b21 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007fe47488d800 nid=0x54b7 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"taskCleanup" daemon prio=10 tid=0x00007fe4749c5000 nid=0x6190 waiting on condition [0x00007fe465fa9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d0370f60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:530)
at java.lang.Thread.run(Thread.java:722)
"Thread-6" daemon prio=10 tid=0x00007fe4749f4000 nid=0x618f waiting on condition [0x00007fe4660aa000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d0370fc0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager.monitor(UserLogManager.java:131)
at org.apache.hadoop.mapreduce.server.tasktracker.userlogs.UserLogManager$1.run(UserLogManager.java:66)
"Thread-7" daemon prio=10 tid=0x00007fe474a0a000 nid=0x618e waiting on condition [0x00007fe4661ab000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapred.UserLogCleaner.run(UserLogCleaner.java:93)
"Timer-0" daemon prio=10 tid=0x00007fe4749cb800 nid=0x618d in Object.wait() [0x00007fe4662ac000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000d03772e0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"149446633@qtp-633284350-0 - Acceptor0 SelectChannelConnector@0.0.0.0:50060" prio=10 tid=0x00007fe47489a800 nid=0x618c runnable [0x00007fe466
3ad000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d0371008> (a sun.nio.ch.Util$2)
- locked <0x00000000d0371020> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d0378a90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:511)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
"Monitor for Jetty bugs" daemon prio=10 tid=0x00007fe474897800 nid=0x618b sleeping[0x00007fe4664ae000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapred.JettyBugMonitor.monitorThreads(JettyBugMonitor.java:142)
at org.apache.hadoop.mapred.JettyBugMonitor.doRun(JettyBugMonitor.java:121)
at org.apache.hadoop.mapred.JettyBugMonitor.run(JettyBugMonitor.java:102)
"TaskLauncher for REDUCE tasks" daemon prio=10 tid=0x00007fe474892000 nid=0x618a in Object.wait() [0x00007fe4665af000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2472)
- locked <0x00000000d0371b00> (a java.util.LinkedList)
"TaskLauncher for MAP tasks" daemon prio=10 tid=0x00007fe474876800 nid=0x6189 in Object.wait() [0x00007fe4666b0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2472)
- locked <0x00000000d0371b20> (a java.util.LinkedList)
"Map-events fetcher for all reduce tasks on tracker_xxxxxx.jp:localhost/127.0.0.1:39752" daemon prio=10 tid=0x00007fe47483c000 nid=0x6182 in Object.wait() [0x00007fe4667b1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.mapred.TaskTracker$MapEventsFetcherThread.run(TaskTracker.java:1094)
- locked <0x00000000d037b790> (a java.util.TreeMap)
"sendParams-0" daemon prio=10 tid=0x00007fe47483a000 nid=0x6181 waiting on condition [0x00007fe4668b2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d03794a8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"IPC Client (47) connection to xxxxx.jp/192.168.171.226:9001 from mapred" daemon prio=10 tid=0x00007fe474837800 nid=0x6180 runnable [0x00007fe4669b3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d057cd48> (a sun.nio.ch.Util$2)
- locked <0x00000000d057cd30> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d0539700> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:375)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
- locked <0x00000000d0712f40> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:815)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:724)
"IPC Server handler 3 on 39752" daemon prio=10 tid=0x00007fe474805800 nid=0x617f waiting on condition [0x00007fe466ab4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d0371080> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
"IPC Server handler 2 on 39752" daemon prio=10 tid=0x00007fe474803800 nid=0x617e waiting on condition [0x00007fe466bb5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d0371080> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
"IPC Server handler 1 on 39752" daemon prio=10 tid=0x00007fe474801800 nid=0x617d waiting on condition [0x00007fe466cb6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d0371080> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
"IPC Server handler 0 on 39752" daemon prio=10 tid=0x00007fe4747ff800 nid=0x617c waiting on condition [0x00007fe466db7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d0371080> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1410)
"IPC Server listener on 39752" daemon prio=10 tid=0x00007fe4747fd800 nid=0x617b runnable [0x00007fe466eb8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d0377358> (a sun.nio.ch.Util$2)
- locked <0x00000000d0377370> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d037fbf0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.hadoop.ipc.Server$Listener.run(Server.java:468)
"IPC Server Responder" daemon prio=10 tid=0x00007fe4747e2000 nid=0x617a runnable [0x00007fe466fb9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d037bad8> (a sun.nio.ch.Util$2)
- locked <0x00000000d037baf0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d0378ad8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.hadoop.ipc.Server$Responder.run(Server.java:638)
"Timer thread for monitoring rpc" daemon prio=10 tid=0x00007fe4746f4800 nid=0x6179 in Object.wait() [0x00007fe4670ba000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000d037dc40> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Socket Reader #1 for port 39752" prio=10 tid=0x00007fe4746f1000 nid=0x6178 runnable [0x00007fe4671bb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d03794d8> (a sun.nio.ch.Util$2)
- locked <0x00000000d03794f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d037be58> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:352)
- locked <0x00000000d0385420> (a org.apache.hadoop.ipc.Server$Listener$Reader)
"Timer thread for monitoring jvm" daemon prio=10 tid=0x00007fe4746a8000 nid=0x6177 in Object.wait() [0x00007fe4672bc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000d037bb08> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Directory/File cleanup thread" daemon prio=10 tid=0x00007fe47469d000 nid=0x6175 waiting on condition [0x00007fe4676c6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d037bb38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.mapred.CleanupQueue$PathCleanupThread.run(CleanupQueue.java:130)
"Timer thread for monitoring mapred" daemon prio=10 tid=0x00007fe474674800 nid=0x616f in Object.wait() [0x00007fe4677c7000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000000d037bb80> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Service Thread" daemon prio=10 tid=0x00007fe4741e6800 nid=0x616b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007fe4741e4000 nid=0x616a waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007fe4741e1800 nid=0x6169 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007fe4741df800 nid=0x6168 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007fe4741dd800 nid=0x6167 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007fe474191800 nid=0x6166 in Object.wait() [0x00007fe46d320000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000d037bbc8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x00007fe47418f000 nid=0x6165 in Object.wait() [0x00007fe46d421000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000d037bbb0> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007fe474009000 nid=0x6158 in Object.wait() [0x00007fe47c1da000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.ipc.Client.call(Client.java:1093)
- locked <0x00000000bcfce3c0> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at org.apache.hadoop.mapred.$Proxy4.heartbeat(Unknown Source)
at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1990)
at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1784)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2636)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3881)
"VM Thread" prio=10 tid=0x00007fe474187800 nid=0x6164 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007fe474012000 nid=0x6159 runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007fe474014000 nid=0x615a runnable
"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007fe474016000 nid=0x615b runnable
"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007fe474017800 nid=0x615c runnable
"Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00007fe474019800 nid=0x615d runnable
"Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00007fe47401b800 nid=0x615e runnable
"Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00007fe47401d000 nid=0x615f runnable
"Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00007fe47401f000 nid=0x6160 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007fe474112800 nid=0x6163 runnable
"Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x00007fe47410e800 nid=0x6161 runnable
"Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x00007fe474110800 nid=0x6162 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007fe4741f1800 nid=0x616c runnable
JNI global references: 259
Heap
par new generation total 115264K, used 35345K [0x00000000bae00000, 0x00000000c2b10000, 0x00000000d0350000)
eden space 102464K, 34% used [0x00000000bae00000, 0x00000000bd024868, 0x00000000c1210000)
from space 12800K, 2% used [0x00000000c1210000, 0x00000000c126fe58, 0x00000000c1e90000)
to space 12800K, 0% used [0x00000000c1e90000, 0x00000000c1e90000, 0x00000000c2b10000)
concurrent mark-sweep generation total 256256K, used 11948K [0x00000000d0350000, 0x00000000dfd90000, 0x00000000fae00000)
concurrent-mark-sweep perm gen total 25280K, used 17535K [0x00000000fae00000, 0x00000000fc6b0000, 0x0000000100000000)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment