Skip to content

Instantly share code, notes, and snippets.

@rednaxelafx

rednaxelafx/log1

Created July 14, 2011 03:49
Show Gist options
  • Star 19 You must be signed in to star a gist
  • Fork 10 You must be signed in to fork a gist
  • Save rednaxelafx/1081908 to your computer and use it in GitHub Desktop.
Save rednaxelafx/1081908 to your computer and use it in GitHub Desktop.
Download ZIP
analyzing an endless loop problem caused by concurrent access to HashMap.get()
Raw
log1
$ top -H
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19115 root 25 0 576m 217m 9432 R 99.7 5.3 1357:12 java
28347 root 25 0 576m 217m 9432 R 99.7 5.3 1191:05 java
26912 root 25 0 576m 217m 9432 R 99.3 5.3 114:58.57 java
5607 root 25 0 576m 217m 9432 R 99.0 5.3 54:49.02 java
Raw
log2
$ sudo -u root jstack 400
2011-07-14 10:27:41
Full thread dump Java HotSpot(TM) Server VM (16.0-b10 mixed mode):
"Attach Listener" daemon prio=10 tid=0x09966400 nid=0x3597 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"pool-1-thread-53" prio=10 tid=0x971f7800 nid=0x29c0 waiting on condition [0x9b7fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xa175ed10> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-51" prio=10 tid=0x9a134400 nid=0x15e7 runnable [0x9c10b000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapEntrySetStore.java:327)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:221)
- locked <0xa26c2110> (a org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore)
at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987)
at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258)
at org.datanucleus.sco.backed.Map.load(Map.java:229)
at org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1609)
at org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(PersistentClean.java:189)
at org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470)
at org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131)
at org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
- locked <0xa2672c58> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
- locked <0xa2672c58> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:618)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:551)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:924)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:558)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:515)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:475)
at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:1838)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:176)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-50" prio=10 tid=0x9a156000 nid=0x6920 runnable [0x9b35c000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapEntrySetStore.java:322)
at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:221)
- locked <0xa2636548> (a org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore)
at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987)
at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258)
at org.datanucleus.sco.backed.Map.load(Map.java:229)
at org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields(JDOStateManagerImpl.java:1609)
at org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(PersistentClean.java:189)
at org.datanucleus.state.AbstractStateManager.retrieve(AbstractStateManager.java:470)
at org.datanucleus.ObjectManagerImpl.retrieveObject(ObjectManagerImpl.java:1131)
at org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:534)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:551)
- locked <0xa2503a60> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.datanucleus.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:560)
- locked <0xa2503a60> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:618)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:551)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:924)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:921)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:558)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:515)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:475)
at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:1838)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:176)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"LeaseChecker" daemon prio=10 tid=0x09af4c00 nid=0x29fd waiting on condition [0x9b3ad000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:978)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-43" prio=10 tid=0x9b6e1000 nid=0x6ebb runnable [0x9bead000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(MappedTypeManager.java:84)
at org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.<init>(AbstractIteratorStatement.java:83)
at org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.<init>(UnionIteratorStatement.java:147)
at org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(ClassTableExtent.java:204)
at org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile(QueryCompiler.java:323)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(JDOQLQueryCompiler.java:225)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(JDOQLQuery.java:175)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1628)
at org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(JDOQLQuery.java:245)
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1499)
at org.datanucleus.jdo.JDOQuery.execute(JDOQuery.java:290)
at org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:826)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartition(ObjectStore.java:797)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(HiveMetaStore.java:1282)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(HiveMetaStore.java:1279)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(HiveMetaStore.java:1279)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:542)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:919)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:282)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:191)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-1-thread-25" prio=10 tid=0x99389c00 nid=0x4aab runnable [0x9c1ac000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:185)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(SQLExpressionFactory.java:159)
at org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getSQLStatementForGet(RDBMSJoinMapStore.java:469)
at org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getValue(RDBMSJoinMapStore.java:320)
- locked <0xa1fed690> (a org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore)
at org.datanucleus.store.mapped.scostore.JoinMapStore.putAll(JoinMapStore.java:167)
at org.datanucleus.store.mapped.mapping.MapMapping.postInsert(MapMapping.java:135)
at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:515)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121)
at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275)
at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)
at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(PersistableMapping.java:664)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(PersistableMapping.java:423)
at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:197)
at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1023)
at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(MStorageDescriptor.java)
at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(MStorageDescriptor.java)
at org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(JDOStateManagerImpl.java:2803)
- locked <0xa1fedaa8> (a java.lang.Object)
at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:294)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121)
at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275)
at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)
at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(PersistableMapping.java:664)
at org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(PersistableMapping.java:423)
at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:197)
at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1023)
at org.apache.hadoop.hive.metastore.model.MTable.jdoProvideField(MTable.java)
at org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFields(MTable.java)
at org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(JDOStateManagerImpl.java:2803)
- locked <0xa1fedc30> (a java.lang.Object)
at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:294)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:146)
at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:121)
at org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent(JDOStateManagerImpl.java:3275)
at org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent(JDOStateManagerImpl.java:3249)
at org.datanucleus.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1428)
at org.datanucleus.ObjectManagerImpl.persistObject(ObjectManagerImpl.java:1241)
at org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:655)
at org.datanucleus.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:680)
- locked <0xa1fedc88> (a org.datanucleus.jdo.JDOPersistenceManager)
at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:515)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:793)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$600(HiveMetaStore.java:79)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(HiveMetaStore.java:815)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(HiveMetaStore.java:812)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:234)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:812)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:279)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:394)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:2126)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:166)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:115)
at org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(ThriftHive.java:378)
at org.apache.hadoop.hive.service.ThriftHive$Processor.process(ThriftHive.java:366)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x082ab800 nid=0x2b0a in Object.wait() [0x9ad9c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa1a09490> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked <0xa1a09490> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Low Memory Detector" daemon prio=10 tid=0x08119000 nid=0x1ef runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x08117800 nid=0x1ee waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x08114400 nid=0x1ed waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x08112c00 nid=0x1ec runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x080ff000 nid=0x1eb in Object.wait() [0x9d0f2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0xa1717c28> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x080fd800 nid=0x1ea in Object.wait() [0x9d143000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0xa1749a58> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0805d400 nid=0x1e4 runnable [0xb7471000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0xa175ebe8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:34)
at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:183)
at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:390)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
"VM Thread" prio=10 tid=0x080f9c00 nid=0x1e9 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x08064400 nid=0x1e5 runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x08065c00 nid=0x1e6 runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x08067400 nid=0x1e7 runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x08068800 nid=0x1e8 runnable
"VM Periodic Task Thread" prio=10 tid=0x0811b000 nid=0x1f0 waiting on condition
JNI global references: 760
Raw
log3
$ sudo -u root jstack -F 400
Attaching to process ID 400, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.0-b10
Deadlock Detection:
No deadlocks found.
Thread 13719: (state = BLOCKED)
Thread 10688: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=158 (Interpreted frame)
- java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=158, line=422 (Compiled frame)
- java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=323 (Interpreted frame)
- java.util.concurrent.SynchronousQueue.take() @bci=7, line=857 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 5607: (state = IN_JAVA)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(org.datanucleus.StateManager) @bci=137, line=327 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(org.datanucleus.StateManager) @bci=20, line=221 (Compiled frame)
- org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(java.util.Map, org.datanucleus.store.scostore.MapStore, org.datanucleus.StateManager) @bci=164, line=987 (Compiled frame)
- org.datanucleus.sco.backed.Map.loadFromStore() @bci=74, line=258 (Compiled frame)
- org.datanucleus.sco.backed.Map.load() @bci=8, line=229 (Compiled frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields() @bci=160, line=1609 (Compiled frame)
- org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(org.datanucleus.StateManager, boolean) @bci=14, line=189 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.retrieve(boolean) @bci=11, line=470 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.retrieveObject(java.lang.Object, boolean) @bci=72, line=1131 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(java.lang.Object, boolean) @bci=6, line=534 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object, boolean) @bci=7, line=551 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object) @bci=3, line=560 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getMTable(java.lang.String, java.lang.String) @bci=77, line=618 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getTable(java.lang.String, java.lang.String) @bci=14, line=551 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=924 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(java.lang.String, java.lang.String) @bci=30, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(java.lang.String, java.lang.String) @bci=7, line=558 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String, java.lang.String, boolean) @bci=32, line=515 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String) @bci=7, line=475 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.DropTableDesc) @bci=7, line=1838 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=152, line=176 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Compiled frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 26912: (state = IN_JAVA)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(org.datanucleus.StateManager) @bci=94, line=322 (Compiled frame)
- org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(org.datanucleus.StateManager) @bci=20, line=221 (Compiled frame)
- org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(java.util.Map, org.datanucleus.store.scostore.MapStore, org.datanucleus.StateManager) @bci=164, line=987 (Compiled frame)
- org.datanucleus.sco.backed.Map.loadFromStore() @bci=74, line=258 (Compiled frame)
- org.datanucleus.sco.backed.Map.load() @bci=8, line=229 (Compiled frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.loadUnloadedFields() @bci=160, line=1609 (Compiled frame)
- org.datanucleus.jdo.state.PersistentClean.transitionRetrieve(org.datanucleus.StateManager, boolean) @bci=14, line=189 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.retrieve(boolean) @bci=11, line=470 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.retrieveObject(java.lang.Object, boolean) @bci=72, line=1131 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.jdoRetrieve(java.lang.Object, boolean) @bci=6, line=534 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object, boolean) @bci=7, line=551 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.retrieve(java.lang.Object) @bci=3, line=560 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getMTable(java.lang.String, java.lang.String) @bci=77, line=618 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getTable(java.lang.String, java.lang.String) @bci=14, line=551 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=924 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$15.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(java.lang.String, java.lang.String) @bci=30, line=921 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(java.lang.String, java.lang.String) @bci=7, line=558 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String, java.lang.String, boolean) @bci=32, line=515 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getTable(java.lang.String) @bci=7, line=475 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.DropTableDesc) @bci=7, line=1838 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=152, line=176 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Compiled frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Compiled frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 10749: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
- org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run() @bci=73, line=978 (Compiled frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 28347: (state = IN_JAVA)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=1, line=185 (Compiled frame; information may be imprecise)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.mapped.MappedTypeManager.isSupportedMappedType(java.lang.String) @bci=21, line=84 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.AbstractIteratorStatement.<init>(java.lang.Class, org.datanucleus.ClassLoaderResolver, boolean, org.datanucleus.store.StoreManager) @bci=88, line=83 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.UnionIteratorStatement.<init>(org.datanucleus.ClassLoaderResolver, java.lang.Class, boolean, org.datanucleus.store.StoreManager, java.lang.Class, org.datanucleus.store.mapped.mapping.JavaTypeMapping, org.datanucleus.store.mapped.DatastoreContainerObject, boolean, java.lang.Boolean, boolean, boolean) @bci=6, line=147 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.ClassTableExtent.newQueryStatement(java.lang.Class, org.datanucleus.store.mapped.DatastoreIdentifier) @bci=343, line=204 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.QueryCompiler.executionCompile() @bci=77, line=323 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.JDOQLQueryCompiler.compile(int) @bci=567, line=225 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.compileInternal(boolean, java.util.Map) @bci=60, line=175 (Interpreted frame)
- org.datanucleus.store.query.Query.executeQuery(java.util.Map) @bci=51, line=1628 (Interpreted frame)
- org.datanucleus.store.rdbms.query.legacy.JDOQLQuery.executeQuery(java.util.Map) @bci=6, line=245 (Interpreted frame)
- org.datanucleus.store.query.Query.executeWithArray(java.lang.Object[]) @bci=102, line=1499 (Interpreted frame)
- org.datanucleus.jdo.JDOQuery.execute(java.lang.Object, java.lang.Object, java.lang.Object) @bci=20, line=290 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(java.lang.String, java.lang.String, java.util.List) @bci=117, line=826 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.getPartition(java.lang.String, java.lang.String, java.util.List) @bci=10, line=797 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(org.apache.hadoop.hive.metastore.RawStore) @bci=13, line=1282 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$20.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=1279 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition(java.lang.String, java.lang.String, java.util.List) @bci=33, line=1279 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(java.lang.String, java.lang.String, java.util.List) @bci=8, line=542 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.getPartition(org.apache.hadoop.hive.ql.metadata.Table, java.util.Map, boolean) @bci=190, line=919 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.AddPartitionDesc) @bci=28, line=282 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=227, line=191 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Interpreted frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 19115: (state = IN_JAVA)
- java.util.HashMap.get(java.lang.Object) @bci=74, line=303 (Compiled frame; information may be imprecise)
- org.datanucleus.util.ReferenceValueMap.get(java.lang.Object) @bci=9, line=186 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String, java.lang.ClassLoader) @bci=195, line=185 (Compiled frame)
- org.datanucleus.JDOClassLoaderResolver.classForName(java.lang.String) @bci=3, line=415 (Compiled frame)
- org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newExpression(org.datanucleus.store.rdbms.sql.SQLStatement, org.datanucleus.store.rdbms.sql.SQLTable, org.datanucleus.store.mapped.mapping.JavaTypeMapping) @bci=63, line=159 (Interpreted frame)
- org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getSQLStatementForGet(org.datanucleus.StateManager) @bci=220, line=469 (Interpreted frame)
- org.datanucleus.store.rdbms.scostore.RDBMSJoinMapStore.getValue(org.datanucleus.StateManager, java.lang.Object) @bci=32, line=320 (Interpreted frame)
- org.datanucleus.store.mapped.scostore.JoinMapStore.putAll(org.datanucleus.StateManager, java.util.Map) @bci=102, line=167 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.MapMapping.postInsert(org.datanucleus.StateManager) @bci=387, line=135 (Interpreted frame)
- org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=1914, line=515 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=689, line=664 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=25, line=423 (Interpreted frame)
- org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(int, java.lang.Object) @bci=160, line=197 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.providedObjectField(javax.jdo.spi.PersistenceCapable, int, java.lang.Object) @bci=6, line=1023 (Compiled frame)
- org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideField(int) @bci=226 (Interpreted frame)
- org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoProvideFields(int[]) @bci=28 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(int[], org.datanucleus.store.fieldmanager.FieldManager) @bci=23, line=2803 (Compiled frame)
- org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=562, line=294 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObjectAsValue(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=689, line=664 (Interpreted frame)
- org.datanucleus.store.mapped.mapping.PersistableMapping.setObject(org.datanucleus.ObjectManager, java.lang.Object, int[], java.lang.Object, org.datanucleus.StateManager, int) @bci=25, line=423 (Interpreted frame)
- org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(int, java.lang.Object) @bci=160, line=197 (Interpreted frame)
- org.datanucleus.state.AbstractStateManager.providedObjectField(javax.jdo.spi.PersistenceCapable, int, java.lang.Object) @bci=6, line=1023 (Compiled frame)
- org.apache.hadoop.hive.metastore.model.MTable.jdoProvideField(int) @bci=216 (Interpreted frame)
- org.apache.hadoop.hive.metastore.model.MTable.jdoProvideFields(int[]) @bci=28 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.provideFields(int[], org.datanucleus.store.fieldmanager.FieldManager) @bci=23, line=2803 (Compiled frame)
- org.datanucleus.store.rdbms.request.InsertRequest.execute(org.datanucleus.StateManager) @bci=562, line=294 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(org.datanucleus.store.mapped.DatastoreClass, org.datanucleus.StateManager, org.datanucleus.ClassLoaderResolver) @bci=75, line=146 (Interpreted frame)
- org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(org.datanucleus.StateManager) @bci=152, line=121 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.internalMakePersistent() @bci=63, line=3275 (Interpreted frame)
- org.datanucleus.jdo.state.JDOStateManagerImpl.makePersistent() @bci=258, line=3249 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObjectInternal(java.lang.Object, org.datanucleus.store.FieldValues, org.datanucleus.StateManager, int, int) @bci=370, line=1428 (Interpreted frame)
- org.datanucleus.ObjectManagerImpl.persistObject(java.lang.Object) @bci=97, line=1241 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.jdoMakePersistent(java.lang.Object) @bci=5, line=655 (Interpreted frame)
- org.datanucleus.jdo.JDOPersistenceManager.makePersistent(java.lang.Object) @bci=16, line=680 (Interpreted frame)
- org.apache.hadoop.hive.metastore.ObjectStore.createTable(org.apache.hadoop.hive.metastore.api.Table) @bci=18, line=515 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(org.apache.hadoop.hive.metastore.RawStore, org.apache.hadoop.hive.metastore.api.Table) @bci=371, line=793 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.access$600(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler, org.apache.hadoop.hive.metastore.RawStore, org.apache.hadoop.hive.metastore.api.Table) @bci=3, line=79 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(org.apache.hadoop.hive.metastore.RawStore) @bci=9, line=815 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$13.run(org.apache.hadoop.hive.metastore.RawStore) @bci=2, line=812 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$Command) @bci=60, line=234 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(org.apache.hadoop.hive.metastore.api.Table) @bci=55, line=812 (Interpreted frame)
- org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(org.apache.hadoop.hive.metastore.api.Table) @bci=24, line=279 (Interpreted frame)
- org.apache.hadoop.hive.ql.metadata.Hive.createTable(org.apache.hadoop.hive.ql.metadata.Table, boolean) @bci=69, line=394 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.createTable(org.apache.hadoop.hive.ql.metadata.Hive, org.apache.hadoop.hive.ql.plan.CreateTableDesc) @bci=771, line=2126 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.DDLTask.execute(org.apache.hadoop.hive.ql.DriverContext) @bci=102, line=166 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.Task.executeTask() @bci=28, line=107 (Interpreted frame)
- org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential() @bci=4, line=55 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.launchTask(org.apache.hadoop.hive.ql.exec.Task, java.lang.String, boolean, java.util.Map, java.lang.String, int, org.apache.hadoop.hive.ql.DriverContext) @bci=194, line=633 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.execute() @bci=475, line=506 (Interpreted frame)
- org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=38, line=384 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(java.lang.String) @bci=99, line=115 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor$execute.process(int, org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=40, line=378 (Interpreted frame)
- org.apache.hadoop.hive.service.ThriftHive$Processor.process(org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol) @bci=120, line=366 (Interpreted frame)
- org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=102, line=252 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=619 (Interpreted frame)
Thread 11018: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- java.util.TimerThread.mainLoop() @bci=28, line=483 (Interpreted frame)
- java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame)
Thread 492: (state = BLOCKED)
Thread 491: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Compiled frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Compiled frame)
Thread 490: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
Thread 484: (state = IN_NATIVE)
- java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=390 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
- org.apache.thrift.transport.TServerSocket.acceptImpl() @bci=22, line=118 (Interpreted frame)
- org.apache.thrift.transport.TServerSocket.acceptImpl() @bci=1, line=34 (Interpreted frame)
- org.apache.thrift.transport.TServerTransport.accept() @bci=1, line=31 (Interpreted frame)
- org.apache.thrift.server.TThreadPoolServer.serve() @bci=39, line=183 (Interpreted frame)
- org.apache.hadoop.hive.service.HiveServer.main(java.lang.String[]) @bci=114, line=390 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Interpreted frame)
- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=161, line=597 (Interpreted frame)
- org.apache.hadoop.util.RunJar.main(java.lang.String[]) @bci=595, line=165 (Interpreted frame)
- org.apache.hadoop.mapred.JobShell.run(java.lang.String[]) @bci=12, line=54 (Interpreted frame)
- org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.conf.Configuration, org.apache.hadoop.util.Tool, java.lang.String[]) @bci=38, line=65 (Interpreted frame)
- org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.util.Tool, java.lang.String[]) @bci=8, line=79 (Interpreted frame)
- org.apache.hadoop.mapred.JobShell.main(java.lang.String[]) @bci=10, line=68 (Interpreted frame)
Raw
log4
$ sudo -u root jstack -m 400 | c++filt
Attaching to process ID 400, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16.0-b10
Deadlock Detection:
No deadlocks found.
----------------- 484 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0x9cb93e19 Java_java_net_PlainSocketImpl_socketAccept + 0x129
0xb43c25aa * java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) bci:0 (Interpreted frame)
0xb43bb08d * java.net.PlainSocketImpl.accept(java.net.SocketImpl) bci:7 line:390 (Interpreted frame)
0xb43bb08d * java.net.ServerSocket.implAccept(java.net.Socket) bci:60 line:453 (Interpreted frame)
0xb43bb08d * java.net.ServerSocket.accept() bci:48 line:421 (Interpreted frame)
0xb43baf27 * org.apache.thrift.transport.TServerSocket.acceptImpl() bci:22 line:118 (Interpreted frame)
0xb43baf27 * org.apache.thrift.transport.TServerSocket.acceptImpl() bci:1 line:34 (Interpreted frame)
0xb43baf27 * org.apache.thrift.transport.TServerTransport.accept() bci:1 line:31 (Interpreted frame)
0xb43baf27 * org.apache.thrift.server.TThreadPoolServer.serve() bci:39 line:183 (Interpreted frame)
0xb43bb08d * org.apache.hadoop.hive.service.HiveServer.main(java.lang.String[]) bci:114 line:390 (Interpreted frame)
0xb43b834c <StubRoutines>
0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190
0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18
0xb77f66cf JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) + 0x2f
0xb7a1d92b Reflection::invoke(instanceKlassHandle, methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) + 0x44b
0xb7a20891 Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*) + 0x111
0xb787267f JVM_InvokeMethod + 0x1bf
0xb7406314 Java_sun_reflect_NativeMethodAccessorImpl_invoke0 + 0x34
0xb43c25aa * sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) bci:0 (Interpreted frame)
0xb43baf27 * sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:87 line:39 (Interpreted frame)
0xb43baf27 * sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) bci:6 line:25 (Interpreted frame)
0xb43bb403 * java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) bci:161 line:597 (Interpreted frame)
0xb43baf27 * org.apache.hadoop.util.RunJar.main(java.lang.String[]) bci:595 line:165 (Interpreted frame)
0xb43bb08d * org.apache.hadoop.mapred.JobShell.run(java.lang.String[]) bci:12 line:54 (Interpreted frame)
0xb43bb445 * org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.conf.Configuration, org.apache.hadoop.util.Tool, java.lang.String[]) bci:38 line:65 (Interpreted frame)
0xb43baf69 * org.apache.hadoop.util.ToolRunner.run(org.apache.hadoop.util.Tool, java.lang.String[]) bci:8 line:79 (Interpreted frame)
0xb43baf69 * org.apache.hadoop.mapred.JobShell.main(java.lang.String[]) bci:10 line:68 (Interpreted frame)
0xb43b834c <StubRoutines>
0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190
0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18
0xb77f66cf JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) + 0x2f
0xb7828c93 jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) + 0x133
0xb7818eec jni_CallStaticVoidMethod + 0x13c
0x08049b98 JavaMain + 0x2c8
0x00623413 start_thread + 0xb3
----------------- 485 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 486 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 487 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 488 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 489 -----------------sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4
0x004a57a2 _dl_sysinfo_int80 at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
+ at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
0x2
0x020cd378 ? at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)?
??????
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 490 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb
0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53
0xb7855a1c JVM_MonitorWait + 0x20c
0xb4522f12 <Unknown compiled code>
0xb488d994 <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x096de19e with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d77d6a8 ????????
----------------- 491 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb
0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53
0xb7855a1c JVM_MonitorWait + 0x20c
0xb4522f12 <Unknown compiled code>
0xb43bb08d * java.lang.ref.ReferenceQueue.remove(long) bci:44 line:118 (Interpreted frame)
0xb483ef14 * java.lang.ref.ReferenceQueue.remove() bci:2 line:134 (Compiled frame)
* java.lang.ref.Finalizer$FinalizerThread.run() bci:3 line:159 (Compiled frame)
sun.jvm.hotspot.debugger.UnmappedAddressException
at sun.jvm.hotspot.debugger.PageCache.checkPage(PageCache.java:208)
at sun.jvm.hotspot.debugger.PageCache.getData(PageCache.java:63)
at sun.jvm.hotspot.debugger.DebuggerBase.readBytes(DebuggerBase.java:217)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:482)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
----------------- 492 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79cb56d os::signal_wait() + 0xd
0xb79c7b2a signal_thread_entry(JavaThread*, Thread*) + 0x7a
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 493 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af29e Monitor::wait(bool, long, bool) + 0xee
0xb76e2ebf CompileQueue::get() + 0x6f
0xb76e4aaa CompileBroker::compiler_thread_loop() + 0x1ea
0xb7ab9586 compiler_thread_entry(JavaThread*, Thread*) + 0x16
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 494 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af29e Monitor::wait(bool, long, bool) + 0xee
0xb76e2ebf CompileQueue::get() + 0x6f
0xb76e4aaa CompileBroker::compiler_thread_loop() + 0x1ea
0xb7ab9586 compiler_thread_entry(JavaThread*, Thread*) + 0x16
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 495 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread*, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb79652bf LowMemoryDetector::low_memory_detector_thread_entry(JavaThread*, Thread*) + 0x9f
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 496 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x00dd7750 ????????
----------------- 11018 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb7a7e0fb ObjectMonitor::wait(long long, bool, Thread*) + 0x5fb
0xb7a7b563 ObjectSynchronizer::wait(Handle, long long, Thread*) + 0x53
0xb7855a1c JVM_MonitorWait + 0x20c
0xb43c25aa * java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xb43bb08d * java.lang.Object.wait() bci:2 line:485 (Interpreted frame)
0xb43bb08d * java.util.TimerThread.mainLoop() bci:28 line:483 (Interpreted frame)
0xb43bb08d * java.util.TimerThread.run() bci:1 line:462 (Interpreted frame)
0xb43b834c <StubRoutines>
0xb77f6870 JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x190
0xb79cdcb8 os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*) + 0x18
0xb77f6077 JavaCalls::call_virtual(JavaValue*, KlassHandle, symbolHandle, symbolHandle, JavaCallArguments*, Thread*) + 0xb7
0xb77f612a JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, symbolHandle, symbolHandle, Thread*) + 0x6a
0xb7876b95 thread_entry(JavaThread*, Thread*) + 0xa5
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 19115 -----------------
0xb45706dd <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 28347 -----------------
0xb45706e0 <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 10749 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x4e1e5493 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x02fe3088 ????????
----------------- 26912 -----------------
0xb45707bf <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 5607 -----------------
0xb45706e0 <Unknown compiled code>
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9d777780 ????????
----------------- 10688 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb79cee18 Parker::park(bool, long long) + 0x2d8
0xb7adb660 Unsafe_Park + 0x1d0
0xb43c25aa * sun.misc.Unsafe.park(boolean, long) bci:0 (Interpreted frame)
0xb43bb08d * java.util.concurrent.locks.LockSupport.park(java.lang.Object) bci:14 line:158 (Interpreted frame)
0xb4994050 * java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) bci:158 line:422 (Compiled frame)
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to read at address: 0x00000005 with alignment: 4
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1.checkAlignment(LinuxDebuggerLocal.java:181)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readCInteger(LinuxDebuggerLocal.java:480)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.readAddress(LinuxDebuggerLocal.java:423)
at sun.jvm.hotspot.debugger.linux.LinuxAddress.getAddressAt(LinuxAddress.java:74)
at sun.jvm.hotspot.debugger.linux.x86.LinuxX86CFrame.sender(LinuxX86CFrame.java:60)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:161)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
at sun.jvm.hotspot.tools.PStack.run(PStack.java:48)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
0x9dbb1848 ????????
----------------- 13719 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0xb75ce9fb AttachListener::dequeue() + 0x7b
0xb75cdb97 attach_listener_thread_entry(JavaThread*, Thread*) + 0x47
0xb7ab2c3e JavaThread::run() + 0x11e
0xb79cf15e java_start(Thread*) + 0x14e
0x00623413 start_thread + 0xb3
----------------- 400 -----------------
0x004a57a2 _dl_sysinfo_int80 + 0x2
0x0804dd78 ContinueInNewThread + 0x88
0x080497f6 main + 0x276
0x004d3df3 __libc_start_main + 0xd3
Raw
notes.md

A HiveServer running on JDK 6 update 18 early access, 32-bit Linux, is suffering from sudden raise of CPU usage. The process' pid is 400.

$ java -version
java version "1.6.0_18-ea"
Java(TM) SE Runtime Environment (build 1.6.0_18-ea-b03)
Java HotSpot(TM) Server VM (build 16.0-b10, mixed mode)
$ uname -a
Linux xxxx.xxx 2.6.9-89.ELxenU #1 SMP Mon Apr 20 10:56:05 EDT 2009 i686 i686 i386 GNU/Linux

The steps taken to analyze the problem:

  1. Use top -H to find out which threads are suffering from high CPU usage problem. See log1. We can target a specific process by using top -p <pid> -H, in this case that'd be top -p 400 -H.
  • Use jstack <pid> to find out what the threads are doing in this program. See log2.
  • The nids in jstack <pid>'s output are hexadecimal representations of the pids from top -H's output (where as jstack -F <pid> and jstack -m <pid> show decimal representation of pid).
  • From the logs collected above, it's clear that the threads in question are:
"pool-1-thread-25" prio=10 tid=0x99389c00 nid=0x4aab runnable [0x9c1ac000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...
"pool-1-thread-43" prio=10 tid=0x9b6e1000 nid=0x6ebb runnable [0x9bead000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...
"pool-1-thread-50" prio=10 tid=0x9a156000 nid=0x6920 runnable [0x9b35c000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...
"pool-1-thread-51" prio=10 tid=0x9a134400 nid=0x15e7 runnable [0x9c10b000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at org.datanucleus.util.ReferenceValueMap.get(ReferenceValueMap.java:186)
        ...

So all 4 threads in question are stuck in an endless loop in HashMap.get():

    public V get(Object key) {
        if (key == null)
            return getForNullKey();
        int hash = hash(key.hashCode());
        for (Entry<K,V> e = table[indexFor(hash, table.length)];
             e != null;
             e = e.next) {
            Object k;
            if (e.hash == hash && ((k = e.key) == key || key.equals(k)))
                return e.value;
        }
        return null;
    }

where line 303 is e = e.next.

Now we can come to the conclusion that the program has hit a known issue of misusing HashMap: unprotected concurrent access to HashMap may cause data to be corrupted, which in turn may cause HashMap.get() to fall into an endless loop.

@fuyou001
Copy link

----------------- 485 -----------------
0x004a57a2 dl_sysinfo_int80 + 0x2
0xb79ce7f5 os::PlatformEvent::park() + 0xe5
0xb79aebce Monitor::IWait(Thread, long long) + 0xee
0xb79af405 Monitor::wait(bool, long, bool) + 0x255
0xb7795b56 GCTaskManager::get_task(unsigned int) + 0x66
0xb7796c9b GCTaskThread::run() + 0xdb
0xb79cf15e java_start(Thread_) + 0x14e
0x00623413 start_thread + 0xb3
问下这些表示什么意思
前面一列是地址，后面的一列看的似懂非懂的，希望可以简单描述下，谢谢!

@rednaxelafx
Copy link
Author

每行都代表一个栈帧。开头的数字是对应的代码的地址，后面是代码的名字（函数名或方法名）加上当前位置的偏移量。

Oracle/Sun的HotSpot VM中每个Java线程只有一个调用栈，所谓的JVM栈与native栈融合在一起了。jstack的-F/-m模式可以将Java与native混合的栈帧全部显示出来，而默认模式则只显示Java栈帧。

@wuzhongxing
Copy link

请教一下：top确定耗cpu的，jstack 确定 at java.util.HashMap.get(HashMap.java:303)存在问题，
jstack -F pid jstack -m pid在分析时有啥用。

@rednaxelafx
Copy link
Author

-F和-m都只是用来顺带演示一下这俩模式的jstack输出跟普通jstack输出的对比而已。没啥特别的。在这个例子里并没有特别的用处。

@wuzhongxing
Copy link

再请教，从top jstack得到的信息看，cpu耗在线程访问hashmap的get方法，如何判断是多个线程对同一个map的访问，并且没有做好同步

@rednaxelafx
Copy link
Author

rednaxelafx commented Jul 24, 2012 via email

@wuzhongxing
Copy link

也碰到这个了，303行

@vongosling
Copy link

Good

@lzrak47m4a1
Copy link

nice

@lazywalson
Copy link

nice

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment