Skip to content

Instantly share code, notes, and snippets.

@ssimeonov

ssimeonov/code.scala

Last active September 24, 2018 06:22
Show Gist options
  • Save ssimeonov/eeb388d13f802689d772 to your computer and use it in GitHub Desktop.
Save ssimeonov/eeb388d13f802689d772 to your computer and use it in GitHub Desktop.
Download ZIP
SPARK-9343: DROP IF EXISTS throws if a table is missing
Raw
code.scala
import org.apache.spark.sql.hive.HiveContext
val ctx = sqlContext.asInstanceOf[HiveContext]
import ctx.implicits._
// Table test is not present
ctx.tableNames
// ERROR Hive: NoSuchObjectException(message:default.test table not found)
ctx.sql("drop table if exists test")
Raw
spark-shell-output.txt
ubuntu@ip-10-88-50-154:~$ ~/spark-1.4.1-bin-hadoop2.6/bin/spark-shell --packages com.databricks:spark-csv_2.10:1.0.3 --driver-memory 52g --conf "spark.driver.extraJavaOptions=-XX:MaxPermSize=512m" --conf "spark.local.dir=/data/spark/tmp"
Ivy Default Cache set to: /home/ubuntu/.ivy2/cache
The jars for the packages stored in: /home/ubuntu/.ivy2/jars
:: loading settings :: url = jar:file:/home/ubuntu/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.databricks#spark-csv_2.10 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.databricks#spark-csv_2.10;1.0.3 in central
found org.apache.commons#commons-csv;1.1 in central
:: resolution report :: resolve 211ms :: artifacts dl 6ms
:: modules in use:
com.databricks#spark-csv_2.10;1.0.3 from central in [default]
org.apache.commons#commons-csv;1.1 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 2 | 0 | 0 | 0 || 2 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 2 already retrieved (0kB/5ms)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/25 15:16:55 INFO SecurityManager: Changing view acls to: ubuntu
15/07/25 15:16:55 INFO SecurityManager: Changing modify acls to: ubuntu
15/07/25 15:16:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); users with modify permissions: Set(ubuntu)
15/07/25 15:16:55 INFO HttpServer: Starting HTTP Server
15/07/25 15:16:56 INFO Utils: Successfully started service 'HTTP class server' on port 50863.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.1
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_79)
Type in expressions to have them evaluated.
Type :help for more information.
15/07/25 15:16:58 WARN Utils: Your hostname, ip-10-88-50-154 resolves to a loopback address: 127.0.0.1; using 10.88.50.154 instead (on interface eth0)
15/07/25 15:16:58 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/07/25 15:16:58 INFO SparkContext: Running Spark version 1.4.1
15/07/25 15:16:58 WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
15/07/25 15:16:59 INFO SecurityManager: Changing view acls to: ubuntu
15/07/25 15:16:59 INFO SecurityManager: Changing modify acls to: ubuntu
15/07/25 15:16:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); users with modify permissions: Set(ubuntu)
15/07/25 15:16:59 INFO Slf4jLogger: Slf4jLogger started
15/07/25 15:16:59 INFO Remoting: Starting remoting
15/07/25 15:16:59 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.88.50.154:34398]
15/07/25 15:16:59 INFO Utils: Successfully started service 'sparkDriver' on port 34398.
15/07/25 15:16:59 INFO SparkEnv: Registering MapOutputTracker
15/07/25 15:16:59 INFO SparkEnv: Registering BlockManagerMaster
15/07/25 15:16:59 INFO DiskBlockManager: Created local directory at /data/spark/tmp/spark-ef8a57c4-2f9e-4c19-aeda-f022637df165/blockmgr-2527b2c6-2576-48cb-8cf8-aab6301cdcc9
15/07/25 15:16:59 INFO MemoryStore: MemoryStore started with capacity 26.9 GB
15/07/25 15:16:59 INFO HttpFileServer: HTTP File server directory is /data/spark/tmp/spark-ef8a57c4-2f9e-4c19-aeda-f022637df165/httpd-11a4291f-3e16-4d75-960f-b8c84339ad5e
15/07/25 15:16:59 INFO HttpServer: Starting HTTP Server
15/07/25 15:16:59 INFO Utils: Successfully started service 'HTTP file server' on port 51912.
15/07/25 15:16:59 INFO SparkEnv: Registering OutputCommitCoordinator
15/07/25 15:16:59 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/07/25 15:16:59 INFO SparkUI: Started SparkUI at http://10.88.50.154:4040
15/07/25 15:16:59 INFO SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/com.databricks_spark-csv_2.10-1.0.3.jar at http://10.88.50.154:51912/jars/com.databricks_spark-csv_2.10-1.0.3.jar with timestamp 1437837419751
15/07/25 15:16:59 INFO SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar at http://10.88.50.154:51912/jars/org.apache.commons_commons-csv-1.1.jar with timestamp 1437837419752
15/07/25 15:16:59 INFO Executor: Starting executor ID driver on host localhost
15/07/25 15:16:59 INFO Executor: Using REPL class URI: http://10.88.50.154:50863
15/07/25 15:16:59 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34739.
15/07/25 15:16:59 INFO NettyBlockTransferService: Server created on 34739
15/07/25 15:16:59 INFO BlockManagerMaster: Trying to register BlockManager
15/07/25 15:16:59 INFO BlockManagerMasterEndpoint: Registering block manager localhost:34739 with 26.9 GB RAM, BlockManagerId(driver, localhost, 34739)
15/07/25 15:16:59 INFO BlockManagerMaster: Registered BlockManager
15/07/25 15:17:00 INFO SparkILoop: Created spark context..
Spark context available as sc.
15/07/25 15:17:00 INFO HiveContext: Initializing execution hive, version 0.13.1
15/07/25 15:17:00 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/07/25 15:17:00 INFO ObjectStore: ObjectStore, initialize called
15/07/25 15:17:00 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/07/25 15:17:00 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/07/25 15:17:00 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/07/25 15:17:01 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/07/25 15:17:02 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/07/25 15:17:02 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
15/07/25 15:17:02 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:02 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:04 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:04 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:04 INFO ObjectStore: Initialized ObjectStore
15/07/25 15:17:04 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/07/25 15:17:05 INFO HiveMetaStore: Added admin role in metastore
15/07/25 15:17:05 INFO HiveMetaStore: Added public role in metastore
15/07/25 15:17:05 INFO HiveMetaStore: No user is added in admin role, since config is empty
15/07/25 15:17:05 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/07/25 15:17:05 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala> import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext
scala>
scala> val ctx = sqlContext.asInstanceOf[HiveContext]
ctx: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@34a0ab9d
scala> import ctx.implicits._
import ctx.implicits._
scala>
scala> // Table test is not present
scala> ctx.tableNames
15/07/25 15:17:27 INFO HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes.
15/07/25 15:17:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/07/25 15:17:28 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/07/25 15:17:28 INFO ObjectStore: ObjectStore, initialize called
15/07/25 15:17:28 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/07/25 15:17:28 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/07/25 15:17:28 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/07/25 15:17:28 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/07/25 15:17:29 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/07/25 15:17:29 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "".
15/07/25 15:17:29 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:29 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:30 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:30 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/07/25 15:17:30 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
15/07/25 15:17:30 INFO ObjectStore: Initialized ObjectStore
15/07/25 15:17:30 INFO HiveMetaStore: Added admin role in metastore
15/07/25 15:17:30 INFO HiveMetaStore: Added public role in metastore
15/07/25 15:17:30 INFO HiveMetaStore: No user is added in admin role, since config is empty
15/07/25 15:17:31 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/07/25 15:17:31 INFO HiveMetaStore: 0: get_tables: db=default pat=.*
15/07/25 15:17:31 INFO audit: ugi=ubuntu ip=unknown-ip-addr cmd=get_tables: db=default pat=.*
res0: Array[String] = Array(agg_origin_by_platform_15m, dimension_components, dimension_creatives, dimension_domains, dimension_keywords, view_clicks, view_clicks_samples)
scala>
scala> // ERROR Hive: NoSuchObjectException(message:default.test table not found)
scala> ctx.sql("drop table if exists test")
15/07/25 15:17:31 INFO ParseDriver: Parsing command: drop table if exists test
15/07/25 15:17:31 INFO ParseDriver: Parse Completed
15/07/25 15:17:31 INFO HiveMetaStore: 0: get_table : db=default tbl=test
15/07/25 15:17:31 INFO audit: ugi=ubuntu ip=unknown-ip-addr cmd=get_table : db=default tbl=test
15/07/25 15:17:31 INFO PerfLogger: <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:31 INFO PerfLogger: <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:31 INFO Driver: Concurrency mode is disabled, not creating a lock manager
15/07/25 15:17:31 INFO PerfLogger: <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:31 INFO PerfLogger: <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:31 INFO ParseDriver: Parsing command: DROP TABLE IF EXISTS test
15/07/25 15:17:32 INFO ParseDriver: Parse Completed
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=parse start=1437837451932 end=1437837452185 duration=253 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO PerfLogger: <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO HiveMetaStore: 0: get_table : db=default tbl=test
15/07/25 15:17:32 INFO audit: ugi=ubuntu ip=unknown-ip-addr cmd=get_table : db=default tbl=test
15/07/25 15:17:32 INFO Driver: Semantic Analysis Completed
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=semanticAnalyze start=1437837452185 end=1437837452222 duration=37 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=compile start=1437837451911 end=1437837452228 duration=317 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO PerfLogger: <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO Driver: Starting command: DROP TABLE IF EXISTS test
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=TimeToSubmit start=1437837451909 end=1437837452243 duration=334 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO PerfLogger: <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO PerfLogger: <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO HiveMetaStore: 0: get_table : db=default tbl=test
15/07/25 15:17:32 INFO audit: ugi=ubuntu ip=unknown-ip-addr cmd=get_table : db=default tbl=test
15/07/25 15:17:32 ERROR Hive: NoSuchObjectException(message:default.test table not found)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1560)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy27.get_table(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy28.getTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:918)
at org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3846)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:345)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$runHive$1.apply(ClientWrapper.scala:326)
at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:155)
at org.apache.spark.sql.hive.client.ClientWrapper.runHive(ClientWrapper.scala:326)
at org.apache.spark.sql.hive.client.ClientWrapper.runSqlHive(ClientWrapper.scala:316)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:473)
at org.apache.spark.sql.hive.execution.DropTable.run(commands.scala:71)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:950)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:950)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:144)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:128)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:755)
at $line25.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:26)
at $line25.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $line25.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33)
at $line25.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35)
at $line25.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:37)
at $line25.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:39)
at $line25.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:41)
at $line25.$read$$iwC$$iwC$$iwC.<init>(<console>:43)
at $line25.$read$$iwC$$iwC.<init>(<console>:45)
at $line25.$read$$iwC.<init>(<console>:47)
at $line25.$read.<init>(<console>:49)
at $line25.$read$.<init>(<console>:53)
at $line25.$read$.<clinit>(<console>)
at $line25.$eval$.<init>(<console>:7)
at $line25.$eval$.<clinit>(<console>)
at $line25.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/07/25 15:17:32 INFO HiveMetaStore: 0: get_table : db=default tbl=test
15/07/25 15:17:32 INFO audit: ugi=ubuntu ip=unknown-ip-addr cmd=get_table : db=default tbl=test
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=runTasks start=1437837452243 end=1437837452305 duration=62 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=Driver.execute start=1437837452228 end=1437837452305 duration=77 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO Driver: OK
15/07/25 15:17:32 INFO PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=releaseLocks start=1437837452309 end=1437837452309 duration=0 from=org.apache.hadoop.hive.ql.Driver>
15/07/25 15:17:32 INFO PerfLogger: </PERFLOG method=Driver.run start=1437837451909 end=1437837452309 duration=400 from=org.apache.hadoop.hive.ql.Driver>
res1: org.apache.spark.sql.DataFrame = []
scala>
@Naman1122
Copy link

Hey did you get resolution for this? I am also facing the same error with Hive in SPark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment