Skip to content

Instantly share code, notes, and snippets.

@fuyou001

fuyou001/log1

Forked from rednaxelafx/log1
Created September 21, 2011 14:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save fuyou001/1232136 to your computer and use it in GitHub Desktop.
Save fuyou001/1232136 to your computer and use it in GitHub Desktop.
Download ZIP
analyzing an endless loop problem caused by concurrent access to HashMap.get()
Raw
log1
$ top -H
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19115 root 25 0 576m 217m 9432 R 99.7 5.3 1357:12 java
28347 root 25 0 576m 217m 9432 R 99.7 5.3 1191:05 java
26912 root 25 0 576m 217m 9432 R 99.3 5.3 114:58.57 java
5607 root 25 0 576m 217m 9432 R 99.0 5.3 54:49.02 java
Raw
log2
$ sudo -u root jstack 400
2011-07-14 10:27:41
Full thread dump Java HotSpot(TM) Server VM (16.0-b10 mixed mode):
"Attach Listener" daemon prio=10 tid=0x09966400 nid=0x3597 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"pool-1-thread-53" prio=10 tid=0x971f7800 nid=0x29c0 waiting on condition [0x9b7fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xa175ed10> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-51" prio=10 tid=0x9a134400 nid=0x15e7 runnable [0x9c10b000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapEntrySetStore.java:327)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:221)
- locked <0xa26c2110> (a org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore)
at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987)
at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258)
at org.datanucleus.sco.backed.Map.load(Map.java:229)
at org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1609)
at org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(PersistentClean.java:189)
at org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470)
at org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131)
at org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
- locked <0xa2672c58> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
- locked <0xa2672c58> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:618)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:551)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:924)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:558)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:515)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:475)
at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:1838)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:176)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-50" prio=10 tid=0x9a156000 nid=0x6920 runnable [0x9b35c000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapEntrySetStore.java:322)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:221)
- locked <0xa2636548> (a org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore)
at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987)
at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258)
at org.datanucleus.sco.backed.Map.load(Map.java:229)
at org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1609)
at org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(PersistentClean.java:189)
at org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470)
at org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131)
at org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
- locked <0xa2503a60> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
- locked <0xa2503a60> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:618)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:551)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:924)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:558)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:515)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:475)
at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:1838)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:176)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"LeaseChecker" daemon prio=10 tid=0x09af4c00 nid=0x29fd waiting on condition [0x9b3ad000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:978)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-43" prio=10 tid=0x9b6e1000 nid=0x6ebb runnable [0x9bead000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(MappedTypeManager.java:84)
at org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.<init>(AbstractIteratorStatement.java:83)
at org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.<init>(UnionIteratorStatement.java:147)
at org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(ClassTableExtent.java:204)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:323)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:290)
at org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:826)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:797)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(HiveMetaStore.java:1282)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(HiveMetaStore.java:1279)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:1279)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:542)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:919)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:282)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:191)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-25" prio=10 tid=0x99389c00 nid=0x4aab runnable [0x9c1ac000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159)
at org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getSQLStatementForGet(RDBMSJoinMapStore.java:469)
at org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getValue(RDBMSJoinMapStore.java:320)
- locked <0xa1fed690> (a org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore)
at org.datanucleus.store.mapped.scostore.JoinMapStore.putAll(JoinMapStore.java:167)
at org.datanucleus.store.mapped.mapping.MapMapping.postInsert(MapMapping.java:135)
at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:515)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121)
at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275)
at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)
at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(PersistableMapping.java:664)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(PersistableMapping.java:423)
at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:197)
at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1023)
at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(MStorageDescriptor.java)
at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(MStorageDescriptor.java)
at org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(JDOStateManagerImpl.java:2803)
- locked <0xa1fedaa8> (a java.lang.Object)
at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:294)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121)
at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275)
at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)
at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(PersistableMapping.java:664)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(PersistableMapping.java:423)
at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:197)
at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1023)
at org.apache.hadoop.hive.metastore.model.MTable.jdoProvideField(MTable.java)
at org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFields(MTable.java)
at org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(JDOStateManagerImpl.java:2803)
- locked <0xa1fedc30> (a java.lang.Object)
at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:294)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121)
at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275)
at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)
at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428)
at org.datanucleus.ObjectManagerImpl.persistObject(ObjectManagerImpl.java:1241)
at org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:655)
at org.datanucleus.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:680)
- locked <0xa1fedc88> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:515)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:793)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$600(HiveMetaStore.java:79)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(HiveMetaStore.java:815)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(HiveMetaStore.java:812)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:812)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:279)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:394)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:2126)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:166)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x082ab800 nid=0x2b0a in Object.wait() [0x9ad9c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa1a09490> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0xa1a09490> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x08119000 nid=0x1ef runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x08117800 nid=0x1ee waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x08114400 nid=0x1ed waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x08112c00 nid=0x1ec runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x080ff000 nid=0x1eb in Object.wait() [0x9d0f2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xa1717c28> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x080fd800 nid=0x1ea in Object.wait() [0x9d143000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0xa1749a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0805d400 nid=0x1e4 runnable [0xb7471000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xa175ebe8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:390)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
"VM Thread" prio=10 tid=0x080f9c00 nid=0x1e9 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x08064400 nid=0x1e5 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x08065c00 nid=0x1e6 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x08067400 nid=0x1e7 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x08068800 nid=0x1e8 runnable
"VM Periodic Task Thread" prio=10 tid=0x0811b000 nid=0x1f0 waiting on condition
JNI global references: 760
Raw
log3
$ sudo -u root jstack -F 400
Attaching to process ID 400, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.0-b10
Deadlock Detection:
No deadlocks found.
Thread 13719: (state = BLOCKED)
Thread 10688: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=158, line=422 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=323 (Interpreted frame)
- java.util.concurrent.SynchronousQueue.take() @bci=7, line=857 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 5607: (state = IN_JAVA)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(org.datanucleus.StateManager) @bci=137, line=327 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(org.datanucleus.StateManager) @bci=20, line=221 (Compiled frame)
- org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(java.util.Map, org.datanucleus.store.scostore.MapStore, org.datanucleus.StateManager) @bci=164, line=987 (Compiled frame)
- org.datanucleus.sco.backed.Map.loadFromStore() @bci=74, line=258 (Compiled frame)
- org.datanucleus.sco.backed.Map.load() @bci=8, line=229 (Compiled frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields() @bci=160, line=1609 (Compiled frame)
- org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(org.datanucleus.StateManager, boolean) @bci=14, line=189 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.retrieve(boolean) @bci=11, line=470 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.retrieveObject(java.lang.Object, boolean) @bci=72, line=1131 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(java.lang.Object, boolean) @bci=6, line=534 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object, boolean) @bci=7, line=551 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object) @bci=3, line=560 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getMTable(java.lang.String, java.lang.String) @bci=77, line=618 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getTable(java.lang.String, java.lang.String) @bci=14, line=551 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=924 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(java.lang.String, java.lang.String) @bci=30, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(java.lang.String, java.lang.String) @bci=7, line=558 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String, java.lang.String, boolean) @bci=32, line=515 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String) @bci=7, line=475 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.DropTableDesc) @bci=7, line=1838 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=152, line=176 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Compiled frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 26912: (state = IN_JAVA)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(org.datanucleus.StateManager) @bci=94, line=322 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(org.datanucleus.StateManager) @bci=20, line=221 (Compiled frame)
- org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(java.util.Map, org.datanucleus.store.scostore.MapStore, org.datanucleus.StateManager) @bci=164, line=987 (Compiled frame)
- org.datanucleus.sco.backed.Map.loadFromStore() @bci=74, line=258 (Compiled frame)
- org.datanucleus.sco.backed.Map.load() @bci=8, line=229 (Compiled frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields() @bci=160, line=1609 (Compiled frame)
- org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(org.datanucleus.StateManager, boolean) @bci=14, line=189 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.retrieve(boolean) @bci=11, line=470 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.retrieveObject(java.lang.Object, boolean) @bci=72, line=1131 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(java.lang.Object, boolean) @bci=6, line=534 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object, boolean) @bci=7, line=551 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object) @bci=3, line=560 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getMTable(java.lang.String, java.lang.String) @bci=77, line=618 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getTable(java.lang.String, java.lang.String) @bci=14, line=551 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=924 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(java.lang.String, java.lang.String) @bci=30, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(java.lang.String, java.lang.String) @bci=7, line=558 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String, java.lang.String, boolean) @bci=32, line=515 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String) @bci=7, line=475 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.DropTableDesc) @bci=7, line=1838 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=152, line=176 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Compiled frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 10749: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run() @bci=73, line=978 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 28347: (state = IN_JAVA)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(java.lang.String) @bci=21, line=84 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.<init>(java.lang.Class, org.datanucleus.ClassLoaderResolver, boolean, org.datanucleus.store.StoreManager) @bci=88, line=83 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.<init>(org.datanucleus.ClassLoaderResolver, java.lang.Class, boolean, org.datanucleus.store.StoreManager, java.lang.Class, org.datanucleus.store.mapped.mapping.JavaTypeMapping, org.datanucleus.store.mapped.DatastoreContainerObject, boolean, java.lang.Boolean, boolean, boolean) @bci=6, line=147 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(java.lang.Class, org.datanucleus.store.mapped.DatastoreIdentifier) @bci=343, line=204 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile() @bci=77, line=323 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(int) @bci=567, line=225 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(boolean, java.util.Map) @bci=60, line=175 (Interpreted frame)
- org.datanucleus.store.query.Query.executeQuery(java.util.Map) @bci=51, line=1628 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(java.util.Map) @bci=6, line=245 (Interpreted frame)
- org.datanucleus.store.query.Query.executeWithArray(java.lang.Object[]) @bci=102, line=1499 (Interpreted frame)
- org.datanucleus.jdo.JDOQuery.execute(java.lang.Object, java.lang.Object, java.lang.Object) @bci=20, line=290 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(java.lang.String, java.lang.String, java.util.List) @bci=117, line=826 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getPartition(java.lang.String, java.lang.String, java.util.List) @bci=10, line=797 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(org.apache.hadoop.hive.metastore.RawStore) @bci=13, line=1282 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=1279 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(java.lang.String, java.lang.String, java.util.List) @bci=33, line=1279 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(java.lang.String, java.lang.String, java.util.List) @bci=8, line=542 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getPartition(org.apache.hadoop.hive.ql.metadata.Table, java.util.Map, boolean) @bci=190, line=919 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.AddPartitionDesc) @bci=28, line=282 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=227, line=191 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Interpreted frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 19115: (state = IN_JAVA)
- java.util.HashMap.get(java.lang.Object) @bci=74, line=303 (Compiled frame; information may be imprecise)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=9, line=186 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Interpreted frame)
- org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getSQLStatementForGet(org.datanucleus.StateManager) @bci=220, line=469 (Interpreted frame)
- org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getValue(org.datanucleus.StateManager, java.lang.Object) @bci=32, line=320 (Interpreted frame)
- org.datanucleus.store.mapped.scostore.JoinMapStore.putAll(org.datanucleus.StateManager, java.util.Map) @bci=102, line=167 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.MapMapping.postInsert(org.datanucleus.StateManager) @bci=387, line=135 (Interpreted frame)
- org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=1914, line=515 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=689, line=664 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=25, line=423 (Interpreted frame)
- org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(int, java.lang.Object) @bci=160, line=197 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.providedObjectField(javax.jdo.spi.PersistenceCapable, int, java.lang.Object) @bci=6, line=1023 (Compiled frame)
- org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(int) @bci=226 (Interpreted frame)
- org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(int[]) @bci=28 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(int[], org.datanucleus.store.fieldmanager.FieldManager) @bci=23, line=2803 (Compiled frame)
- org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=562, line=294 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=689, line=664 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=25, line=423 (Interpreted frame)
- org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(int, java.lang.Object) @bci=160, line=197 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.providedObjectField(javax.jdo.spi.PersistenceCapable, int, java.lang.Object) @bci=6, line=1023 (Compiled frame)
- org.apache.hadoop.hive.metastore.model.MTable.jdoProvideField(int) @bci=216 (Interpreted frame)
- org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFields(int[]) @bci=28 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(int[], org.datanucleus.store.fieldmanager.FieldManager) @bci=23, line=2803 (Compiled frame)
- org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=562, line=294 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObject(java.lang.Object) @bci=97, line=1241 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent(java.lang.Object) @bci=5, line=655 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.makePersistent(java.lang.Object) @bci=16, line=680 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.createTable(org.apache.hadoop.hive.metastore.api.Table) @bci=18, line=515 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(org.apache.hadoop.hive.metastore.RawStore, org.apache.hadoop.hive.metastore.api.Table) @bci=371, line=793 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$600(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler, org.apache.hadoop.hive.metastore.RawStore, org.apache.hadoop.hive.metastore.api.Table) @bci=3, line=79 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=815 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=812 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(org.apache.hadoop.hive.metastore.api.Table) @bci=55, line=812 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(org.apache.hadoop.hive.metastore.api.Table) @bci=24, line=279 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.createTable(org.apache.hadoop.hive.ql.metadata.Table, boolean) @bci=69, line=394 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.createTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.CreateTableDesc) @bci=771, line=2126 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=102, line=166 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Interpreted frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 11018: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=28, line=483 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame)
Thread 492: (state = BLOCKED)
Thread 491: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame)
Thread 490: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
Thread 484: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=390 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
- org.apache.thrift.transport.TServerSocket.acceptImpl() @bci=22, line=118 (Interpreted frame)
- org.apache.thrift.transport.TServerSocket.acceptImpl() @bci=1, line=34 (Interpreted frame)
- org.apache.thrift.transport.TServerTransport.accept() @bci=1, line=31 (Interpreted frame)
- org.apache.thrift.server.TThreadPoolServer.serve() @bci=39, line=183 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer.main(java.lang.String[]) @bci=114, line=390 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame)
- org.apache.hadoop.util.RunJar.main(java.lang.String[]) @bci=595, line=165 (Interpreted frame)
- org.apache.hadoop.mapred.JobShell.run(java.lang.String[]) @bci=12, line=54 (Interpreted frame)
- org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.conf.Configuration, org.apache.hadoop.util.Tool, java.lang.String[]) @bci=38, line=65 (Interpreted frame)
- org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.util.Tool, java.lang.String[]) @bci=8, line=79 (Interpreted frame)
- org.apache.hadoop.mapred.JobShell.main(java.lang.String[]) @bci=10, line=68 (Interpreted frame)
Raw
log4
$ sudo -u root jstack -m 400 | c++filt
Attaching to process ID 400, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.0-b10
Deadlock Detection:
No deadlocks found.
----------------- 484 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0x9cb93e19 Java_java_net_PlainSocketImpl_socketAccept + 0x129
0xb43c25aa * java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) bci:0 (Interpreted frame)
0xb43bb08d * java.net.PlainSocketImpl.accept(java.net.SocketImpl) bci:7 line:390 (Interpreted frame)
0xb43bb08d * java.net.ServerSocket.implAccept(java.net.Socket) bci:60 line:453 (Interpreted frame)
0xb43bb08d * java.net.ServerSocket.accept() bci:48 line:421 (Interpreted frame)
0xb43baf27 * org.apache.thrift.transport.TServerSocket.acceptImpl() bci:22 line:118 (Interpreted frame)
0xb43baf27 * org.apache.thrift.transport.TServerSocket.acceptImpl() bci:1 line:34 (Interpreted frame)
0xb43baf27 * org.apache.thrift.transport.TServerTransport.accept() bci:1 line:31 (Interpreted frame)
0xb43baf27 * org.apache.thrift.server.TThreadPoolServer.serve() bci:39 line:183 (Interpreted frame)
0xb43bb08d * org.apache.hadoop.hive.service.HiveServer.main(java.lang.String[]) bci:114 line:390 (Interpreted frame)
0xb43b834c <StubRoutines>
0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190
0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18
0xb77f66cf JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) + 0x2f
0xb7a1d92b Reflection::invoke(instanceKlassHandle, methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) + 0x44b
0xb7a20891 Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*) + 0x111
0xb787267f JVM_InvokeMethod + 0x1bf
0xb7406314 Java_sun_reflect_NativeMethodAccessorImpl_invoke0 + 0x34
0xb43c25aa * sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) bci:0 (Interpreted frame)
0xb43baf27 * sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:87 line:39 (Interpreted frame)
0xb43baf27 * sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:6 line:25 (Interpreted frame)
0xb43bb403 * java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) bci:161 line:597 (Interpreted frame)
0xb43baf27 * org.apache.hadoop.util.RunJar.main(java.lang.String[]) bci:595 line:165 (Interpreted frame)
0xb43bb08d * org.apache.hadoop.mapred.JobShell.run(java.lang.String[]) bci:12 line:54 (Interpreted frame)
0xb43bb445 * org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.conf.Configuration, org.apache.hadoop.util.Tool, java.lang.String[]) bci:38 line:65 (Interpreted frame)
0xb43baf69 * org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.util.Tool, java.lang.String[]) bci:8 line:79 (Interpreted frame)
0xb43baf69 * org.apache.hadoop.mapred.JobShell.main(java.lang.String[]) bci:10 line:68 (Interpreted frame)
0xb43b834c <StubRoutines>
0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190
0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18
0xb77f66cf JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) + 0x2f
0xb7828c93 jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) + 0x133
0xb7818eec jni_CallStaticVoidMethod + 0x13c
0x08049b98 JavaMain + 0x2c8
0x00623413 start_thread + 0xb3
----------------- 485 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 486 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 487 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 488 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 489 -----------------sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4
0x004a57a2 _dl_sysinfo_int80 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
+ at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
0x2
0x020cd378 ? at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)?
??????
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 490 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb
0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53
0xb7855a1c JVM_MonitorWait + 0x20c
0xb4522f12 <Unknown compiled code>
0xb488d994 <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x096de19e with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d77d6a8 ????????
----------------- 491 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb
0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53
0xb7855a1c JVM_MonitorWait + 0x20c
0xb4522f12 <Unknown compiled code>
0xb43bb08d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:118 (Interpreted frame)
0xb483ef14 * java.lang.ref.ReferenceQueue.remove() bci:2 line:134 (Compiled frame)
* java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159 (Compiled frame)
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 492 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79cb56d os::signal_wait() + 0xd
0xb79c7b2a signal_thread_entry(JavaThread*, Thread*) + 0x7a
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 493 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af29e Monitor::wait(bool, long, bool) + 0xee
0xb76e2ebf CompileQueue::get() + 0x6f
0xb76e4aaa CompileBroker::compiler_thread_loop() + 0x1ea
0xb7ab9586 compiler_thread_entry(JavaThread*, Thread*) + 0x16
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 494 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af29e Monitor::wait(bool, long, bool) + 0xee
0xb76e2ebf CompileQueue::get() + 0x6f
0xb76e4aaa CompileBroker::compiler_thread_loop() + 0x1ea
0xb7ab9586 compiler_thread_entry(JavaThread*, Thread*) + 0x16
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 495 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb79652bf LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*, Thread*) + 0x9f
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 496 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x00dd7750 ????????
----------------- 11018 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb
0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53
0xb7855a1c JVM_MonitorWait + 0x20c
0xb43c25aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xb43bb08d * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xb43bb08d * java.util.TimerThread.mainLoop() bci:28 line:483 (Interpreted frame)
0xb43bb08d * java.util.TimerThread.run() bci:1 line:462 (Interpreted frame)
0xb43b834c <StubRoutines>
0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190
0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18
0xb77f6077 JavaCalls::call_virtual(JavaValue*, KlassHandle, symbolHandle, symbolHandle, JavaCallArguments*, Thread*) + 0xb7
0xb77f612a JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, symbolHandle, symbolHandle, Thread*) + 0x6a
0xb7876b95 thread_entry(JavaThread*, Thread*) + 0xa5
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 19115 -----------------
0xb45706dd <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 28347 -----------------
0xb45706e0 <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 10749 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x02fe3088 ????????
----------------- 26912 -----------------
0xb45707bf <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 5607 -----------------
0xb45706e0 <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 10688 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79cee18 Parker::park(bool, long long) + 0x2d8
0xb7adb660 Unsafe_Park + 0x1d0
0xb43c25aa * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0xb43bb08d * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:158 (Interpreted frame)
0xb4994050 * java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:158 line:422 (Compiled frame)
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9dbb1848 ????????
----------------- 13719 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb75ce9fb AttachListener::dequeue() + 0x7b
0xb75cdb97 attach_listener_thread_entry(JavaThread*, Thread*) + 0x47
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 400 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0x0804dd78 ContinueInNewThread + 0x88
0x080497f6 main + 0x276
0x004d3df3 __libc_start_main + 0xd3
Raw
notes.md

A HiveServer running on JDK 6 update 18 early access, 32-bit Linux, is suffering from sudden raise of CPU usage. The process' pid is 400.

$ java -version
java version "1.6.0_18-ea"
Java(TM) SE Runtime Environment (build 1.6.0_18-ea-b03)
Java HotSpot(TM) Server VM (build 16.0-b10, mixed mode)
$ uname -a
Linux xxxx.xxx 2.6.9-89.ELxenU #1 SMP Mon Apr 20 10:56:05 EDT 2009 i686 i686 i386 GNU/Linux

The steps taken to analyze the problem:

  1. Use top -H to find out which threads are suffering from high CPU usage problem. See log1. We can target a specific process by using top -p <pid> -H, in this case that'd be top -p 400 -H.
  • Use jstack <pid> to find out what the threads are doing in this program. See log2.
  • The nids in jstack <pid>'s output are hexadecimal representations of the pids from top -H's output (where as jstack -F <pid> and jstack -m <pid> show decimal representation of pid).
  • From the logs collected above, it's clear that the threads in question are:
"pool-1-thread-25" prio=10 tid=0x99389c00 nid=0x4aab runnable [0x9c1ac000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...
"pool-1-thread-43" prio=10 tid=0x9b6e1000 nid=0x6ebb runnable [0x9bead000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...
"pool-1-thread-50" prio=10 tid=0x9a156000 nid=0x6920 runnable [0x9b35c000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...
"pool-1-thread-51" prio=10 tid=0x9a134400 nid=0x15e7 runnable [0x9c10b000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...

So all 4 threads in question are stuck in an endless loop in HashMap.get():

    public V get(Object key) {
        if (key == null)
            return getForNullKey();
        int hash = hash(key.hashCode());
        for (Entry<K,V> e = table[indexFor(hash, table.length)];
             e != null;
             e = e.next) {
            Object k;
            if (e.hash == hash && ((k = e.key) == key || key.equals(k)))
                return e.value;
        }
        return null;
    }

where line 303 is e = e.next.

Now we can come to the conclusion that the program has hit a known issue of misusing HashMap: unprotected concurrent access to HashMap may cause data to be corrupted, which in turn may cause HashMap.get() to fall into an endless loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment