Skip to content

Instantly share code, notes, and snippets.

@wyukawa

wyukawa/gist:6fc090985a059136ab6fcfef00790bf9

Last active April 6, 2018 13:53
Show Gist options
  • Save wyukawa/6fc090985a059136ab6fcfef00790bf9 to your computer and use it in GitHub Desktop.
Save wyukawa/6fc090985a059136ab6fcfef00790bf9 to your computer and use it in GitHub Desktop.
Download ZIP
HDP-2.6.2.0-205 resource manager log
Raw
gistfile1.txt
org.apache.hadoop.service.ServiceStateException: Service Registry is in wrong state: STOPPED
at org.apache.hadoop.registry.client.impl.zk.CuratorService.checkServiceLive(CuratorService.java:188)
at org.apache.hadoop.registry.client.impl.zk.CuratorService.zkStat(CuratorService.java:446)
at org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.stat(RegistryOperationsService.java:137)
at org.apache.hadoop.registry.server.services.RegistryAdminService.purge(RegistryAdminService.java:459)
at org.apache.hadoop.registry.server.services.RegistryAdminService.purge(RegistryAdminService.java:520)
at org.apache.hadoop.registry.server.services.RegistryAdminService.purge(RegistryAdminService.java:520)
at org.apache.hadoop.registry.server.services.RegistryAdminService$AsyncPurge.call(RegistryAdminService.java:570)
at org.apache.hadoop.registry.server.services.RegistryAdminService$AsyncPurge.call(RegistryAdminService.java:543)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-04-06 12:24:05,434 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
@wyukawa
Copy link
Author

wyukawa commented Apr 6, 2018

$ grep "Size of event-queue" ...
....
2018-04-06 12:23:02,050 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 91000
2018-04-06 12:23:02,052 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 92000
2018-04-06 12:23:03,052 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 93000
2018-04-06 12:23:03,054 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 94000
2018-04-06 12:23:04,055 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 95000
2018-04-06 12:23:05,056 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 96000
2018-04-06 12:23:08,053 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 97000
2018-04-06 12:23:11,373 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 98000
2018-04-06 12:23:12,545 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 100000
2018-04-06 12:23:12,547 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 101000
2018-04-06 12:23:13,547 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 100000
2018-04-06 12:23:13,550 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 101000
2018-04-06 12:23:14,547 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 102000
2018-04-06 12:23:15,550 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 104000
2018-04-06 12:23:15,796 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 105000
2018-04-06 12:23:16,553 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 106000
2018-04-06 12:23:20,495 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 107000
2018-04-06 12:23:20,497 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 108000
2018-04-06 12:23:21,497 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 109000
2018-04-06 12:23:21,499 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 110000
2018-04-06 12:23:22,499 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 111000
2018-04-06 12:23:23,496 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 112000
2018-04-06 12:23:23,499 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 113000
2018-04-06 12:23:23,505 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 114000
2018-04-06 12:23:24,503 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 115000
2018-04-06 12:23:25,500 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 116000
2018-04-06 12:23:25,509 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 117000
2018-04-06 12:23:26,499 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 118000
2018-04-06 12:23:26,505 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 119000
2018-04-06 12:23:27,562 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 120000
2018-04-06 12:23:34,282 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 121000
2018-04-06 12:23:34,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 122000
2018-04-06 12:23:34,285 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 123000
2018-04-06 12:23:35,286 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 124000
2018-04-06 12:23:36,287 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 125000
2018-04-06 12:23:37,283 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 126000
2018-04-06 12:23:37,286 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 127000
2018-04-06 12:23:37,292 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 128000
2018-04-06 12:23:38,289 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 129000
2018-04-06 12:23:39,287 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 130000
2018-04-06 12:23:39,291 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 131000
2018-04-06 12:23:40,286 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 132000
2018-04-06 12:23:40,292 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 133000
2018-04-06 12:23:53,551 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 124000
2018-04-06 12:23:53,553 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 125000
2018-04-06 12:23:54,550 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 101000
2018-04-06 12:23:56,551 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 102000
2018-04-06 12:23:58,182 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 103000
2018-04-06 12:23:58,227 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 80000
2018-04-06 12:23:58,324 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 59000
2018-04-06 12:23:58,409 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 60000
2018-04-06 12:24:02,140 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 61000
2018-04-06 12:24:02,271 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 62000
2018-04-06 12:24:02,419 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 63000
2018-04-06 12:24:02,505 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 64000
2018-04-06 12:24:02,598 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 65000
2018-04-06 12:24:03,263 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 66000
2018-04-06 12:24:03,456 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 67000
2018-04-06 12:24:03,900 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 68000
2018-04-06 12:24:04,552 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 69000
2018-04-06 12:24:05,262 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Size of event-queue is 70000
...

@wyukawa
Copy link
Author

wyukawa commented Apr 6, 2018 •
edited
Loading

よくみたらその前にzookeeperとの接続が切れてた。。
https://issues.apache.org/jira/browse/YARN-2368
かな

2018-04-06 12:23:58,182 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore: Exception while executing a ZK operation.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /rmstore/ZKRMStateRoot/RMAppRoot/application_1522917813079_16526/appattempt_1522917813079_16526_000001
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$10.run(ZKRMStateStore.java:1092)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$10.run(ZKRMStateStore.java:1089)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1174)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1207)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.existsWithRetries(ZKRMStateStore.java:1089)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.updateApplicationAttemptStateInternal(ZKRMStateStore.java:698)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:317)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$UpdateAppAttemptTransition.transition(RMStateStore.java:299)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.handleStoreEvent(RMStateStore.java:960)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1041)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler.handle(RMStateStore.java:1036)
at org.apache.hadoop.yarn.event.AsyncDispatcher$MultiListenerHandler.handle(AsyncDispatcher.java:279)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
at java.lang.Thread.run(Thread.java:748)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment