Skip to content

Instantly share code, notes, and snippets.

@danieltahara
Created September 23, 2019 17:07
Show Gist options
  • Save danieltahara/dddc778a60cbf8736e32b2a6d978bc35 to your computer and use it in GitHub Desktop.
Save danieltahara/dddc778a60cbf8736e32b2a6d978bc35 to your computer and use it in GitHub Desktop.
hbase
"error_msg": "IOError(message=b\"org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:\\nMon Sep 23 17:01:53 UTC 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60108: Call to ip-172-31-36-201.us-west-2.compute.internal/172.31.36.201:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1070690, waitTime=60008, rpcTimetout=60000 row '414913331#1569257100000' on table 'source_native' at region=source_native,4052#153632595,1565847349122.5dae836fc0cf76b2ebcf59f25c46ca83., hostname=ip-172-31-36-201.us-west-2.compute.internal,16020,1569104584186, seqNum=1697883\\n\\n\\tat org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)\\n\\tat org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)\\n\\tat org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)\\n\\tat org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)\\n\\tat org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)\\n\\tat org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)\\n\\tat org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)\\n\\tat org.apache.hadoop.hbase.client.AbstractClientScanner.next(AbstractClientScanner.java:68)\\n\\tat org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerGetList(ThriftServerRunner.java:1452)\\n\\tat sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)\\n\\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n\\tat java.lang.reflect.Method.invoke(Method.java:498)\\n\\tat org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67)\\n\\tat com.sun.proxy.$Proxy10.scannerGetList(Unknown Source)\\n\\tat org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerGetList.getResult(Hbase.java:4789)\\n\\tat org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerGetList.getResult(Hbase.java:4773)\\n\\tat org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)\\n\\tat org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)\\n\\tat org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:290)\\n\\tat org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)\\n\\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\\n\\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\\n\\tat java.lang.Thread.run(Thread.java:748)\\nCaused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60108: Call to ip-172-31-36-201.us-west-2.compute.internal/172.31.36.201:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1070690, waitTime=60008, rpcTimetout=60000 row '414913331#1569257100000' on table 'source_native' at region=source_native,4052#153632595,1565847349122.5dae836fc0cf76b2ebcf59f25c46ca83., hostname=ip-172-31-36-201.us-west-2.compute.internal,16020,1569104584186, seqNum=1697883\\n\\tat org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)\\n\\tat org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)\\n\\t... 3 more\\nCaused by: java.io.IOException: Call to ip-172-31-36-201.us-west-2.compute.internal/172.31.36.201:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1070690, waitTime=60008, rpcTimetout=60000\\n\\tat org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)\\n\\tat org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)\\n\\tat org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)\\n\\tat org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)\\n\\tat org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)\\n\\tat org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)\\n\\tat org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)\\n\\tat io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)\\n\\tat io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)\\n\\tat io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)\\n\\t... 1 more\\nCaused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=1070690, waitTime=60008, rpcTimetout=60000\\n\\tat org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:189)\\n\\t... 4 more\\n\")",
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment