Skip to content

Instantly share code, notes, and snippets.

@nfarah86

nfarah86/incremental_query.scala

Created April 19, 2023 05:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nfarah86/316e8731246f9c5fd793cd6e51869243 to your computer and use it in GitHub Desktop.
Save nfarah86/316e8731246f9c5fd793cd6e51869243 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
incremental_query.scala
val basePath = "/tmp/hudi/stocks"
spark.read.format("hudi").load(basePath).createOrReplaceTempView("hudi_stocks")
val firstCommit = spark.
sql("select distinct(_hoodie_commit_time) as commitTime from hudi_stocks order by commitTime").
map(k => k.getString(0)).take(1)(0)
val incrementalDF = spark.read.format("hudi").
option("hoodie.datasource.query.type", "incremental").
option("hoodie.datasource.read.begin.instanttime", firstCommit).
load(basePath)
incrementalDF.show()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment