Skip to content

Instantly share code, notes, and snippets.

@mykidong
Created April 24, 2020 06:51
Show Gist options
  • Save mykidong/42ccf87e28309b6e8acb91922e41c878 to your computer and use it in GitHub Desktop.
Save mykidong/42ccf87e28309b6e8acb91922e41c878 to your computer and use it in GitHub Desktop.
Dataset<Row> hiveDf = spark.read().format("hive-with-jdbc")
.option("dbTable", "mc.crawl_youtube")
.option("conditionClause", "where year = '2020' and month = '02' and day = '19'")
.option("hiveJdbcUrl", "jdbc:hive2://mc-d01.mykidong.io:10000")
.option("hiveJdbcUser", "xxxx")
.option("hiveJdbcPassword", "xxxx")
.option("hiveMetastoreUrl", "jdbc:mysql://mc-d01.mykidong.io:3306/hive")
.option("hiveMetastoreUser", "xxxx")
.option("hiveMetastorePassword", "xxxx")
.option("defaultFs", "")
.option("hadoopConfProperties", "hadoop-conf.properties")
.option("outputPath", "/temp-hive")
.load();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment