Skip to content

Instantly share code, notes, and snippets.

@thejasmn

thejasmn/gist:7e33ad38e7bc552e3a659adc041ba200 Secret

Created March 22, 2018 18:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save thejasmn/7e33ad38e7bc552e3a659adc041ba200 to your computer and use it in GitHub Desktop.
Save thejasmn/7e33ad38e7bc552e3a659adc041ba200 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
2018-03-22T17:09:40,855 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ql.Driver (:()) - Compiling command(queryId=hive_20180322170940_c8b75599-531a-4c93-8c35-a651b13b3120): INSERT INTO `VERSION`
VALUES (1, '3.0.0', 'Hive release version 3.0.0')
2018-03-22T17:09:40,860 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: lockmgr.DbTxnManager (:()) - Started heartbeat with delay/interval = 11823/150000 MILLISECONDS for query: null
2018-03-22T17:09:40,860 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Starting Semantic Analysis
2018-03-22T17:09:40,862 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Completed phase 1 of Semantic Analysis
2018-03-22T17:09:40,862 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for source tables
2018-03-22T17:09:40,862 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for subqueries
2018-03-22T17:09:40,862 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for destination tables
2018-03-22T17:09:40,894 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Completed getting MetaData in Semantic Analysis
2018-03-22T17:09:40,916 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ql.Context (:()) - New scratch dir is hdfs://mycluster/tmp/hive/hive/946f78ac-24e9-493b-b877-336d3006a9b5/hive_2018-03-22_17-
09-40_855_212407919803206720-2
2018-03-22T17:09:41,901 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for source tables
2018-03-22T17:09:41,901 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for subqueries
2018-03-22T17:09:41,901 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for source tables
2018-03-22T17:09:41,906 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for subqueries
2018-03-22T17:09:41,906 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for destination tables
2018-03-22T17:09:41,906 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for destination tables
2018-03-22T17:09:41,941 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ql.Context (:()) - New scratch dir is hdfs://mycluster/tmp/hive/hive/946f78ac-24e9-493b-b877-336d3006a9b5/hive_2018-03-22_17-
09-40_855_212407919803206720-2
2018-03-22T17:09:42,160 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Generate an operator pipeline to autogather column stats for table sys.version in query INSERT I
NTO `VERSION` VALUES (1, '3.0.0', 'Hive release version 3.0.0')
2018-03-22T17:09:42,293 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for source tables
2018-03-22T17:09:42,423 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for subqueries
2018-03-22T17:09:42,423 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - Get metadata for destination tables
2018-03-22T17:09:42,433 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ql.Context (:()) - New scratch dir is hdfs://mycluster/tmp/hive/hive/946f78ac-24e9-493b-b877-336d3006a9b5/hive_2018-03-22_17-
09-42_160_6363754285312016946-2
2018-03-22T17:09:42,478 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: parse.CalcitePlanner (:()) - CBO Succeeded; optimized logical plan.
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for FS(5)
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for FS(6)
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for SEL(5)
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for GBY(4)
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for RS(3)
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for GBY(2)
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for SEL(1)
2018-03-22T17:09:42,479 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for SEL(4)
2018-03-22T17:09:42,480 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for SEL(3)
2018-03-22T17:09:42,480 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for UDTF(2)
2018-03-22T17:09:42,480 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for SEL(1)
2018-03-22T17:09:42,480 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: ppd.OpProcFactory (:()) - Processing for TS(0)
2018-03-22T17:09:42,484 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: optimizer.ColumnPrunerProcFactory (:()) - RS 3 oldColExprMap: {VALUE._col2=Column[_col2], VALUE._col0=Column[_col0], VALUE._c
ol1=Column[_col1]}
2018-03-22T17:09:42,484 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: optimizer.ColumnPrunerProcFactory (:()) - RS 3 newColExprMap: {VALUE._col2=Column[_col2], VALUE._col0=Column[_col0], VALUE._c
ol1=Column[_col1]}
2018-03-22T17:09:42,486 INFO [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: correlation.AbstractCorrelationProcCtx (:()) - Overriding hive.optimize.reducededuplication.min.reducer to 1 due to a write t
o transactional table(s) sys.version
2018-03-22T17:09:42,680 ERROR [946f78ac-24e9-493b-b877-336d3006a9b5 HiveServer2-Handler-Pool: Thread-87]: stats.StatsUtils (:()) - Failed to retrieve table statistics:
org.apache.hadoop.hive.ql.metadata.HiveException: NoSuchObjectException(message:Specified database/table does not exist : _dummy_database._dummy_table)
at org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:4163) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:1122) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:342) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:159) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:147) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory$TableScanStatsRule.process(StatsRulesProcFactory.java:137) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.lib.LevelOrderWalker.walk(LevelOrderWalker.java:143) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.lib.LevelOrderWalker.startWalking(LevelOrderWalker.java:122) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.optimizer.stats.annotation.AnnotateWithStatistics.transform(AnnotateWithStatistics.java:78) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsAnnotation(TezCompiler.java:420) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:155) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:144) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11898) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:304) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:280) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:616) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1570) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1523) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198) ~[hive-service-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:284) ~[hive-service-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:243) ~[hive-service-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:541) ~[hive-service-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:527) ~[hive-service-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:311) ~[hive-service-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:564) ~[hive-service-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:647) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.apache.hadoop.hive.metastore.api.NoSuchObjectException: Specified database/table does not exist : _dummy_database._dummy_table
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result$get_table_statistics_req_resultStandardScheme.read(ThriftHiveMetastore.java) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_statistics_req_result.read(ThriftHiveMetastore.java) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_statistics_req(ThriftHiveMetastore.java:3767) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_statistics_req(ThriftHiveMetastore.java:3754) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1738) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableColumnStatistics(SessionHiveMetaStoreClient.java:386) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at com.sun.proxy.$Proxy56.getTableColumnStatistics(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2409) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
at com.sun.proxy.$Proxy56.getTableColumnStatistics(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:4160) ~[hive-exec-3.0.0.3.0.0.0-1076.jar:3.0.0.3.0.0.0-1076]
... 37 more
@jcabasares
Copy link

Got the same issue in hive 'LOAD IN PATH' query. Though the query sometimes proceeds and fails only on this LOAD query. Other queries are running fine. Does someone know how to resolve this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment