Skip to content

Instantly share code, notes, and snippets.

@vinodkc
Last active November 15, 2020 13:56
Show Gist options
  • Save vinodkc/eeb36873b5e659533e1e9120674108be to your computer and use it in GitHub Desktop.
Save vinodkc/eeb36873b5e659533e1e9120674108be to your computer and use it in GitHub Desktop.

How to use hive builtin udf in spark sql

./spark-shell --jars /usr/hdp/current/hive-server2/lib/hive-exec.jar
val data = (1 to 10).toDF("col1").withColumn("col2",col("col1")).registerTempTable("table1")
spark.sql("CREATE TEMPORARY FUNCTION genericUDFAbsFromHive AS 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFAbs'")
sql("select genericUDFAbsFromHive(col1-2000) as absCol1,col2 from table1").show(false)

Output

+-------+----+
|absCol1|col2|
+-------+----+
|1999   |1   |
|1998   |2   |
|1997   |3   |
|1996   |4   |
|1995   |5   |
|1994   |6   |
|1993   |7   |
|1992   |8   |
|1991   |9   |
|1990   |10  |
+-------+----+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment