Skip to content

Instantly share code, notes, and snippets.

@yjshen

yjshen/Construct a batch source. Secret

Created July 15, 2019 07:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yjshen/7853e675604c3690353ef406b61fc7ae to your computer and use it in GitHub Desktop.
Save yjshen/7853e675604c3690353ef406b61fc7ae to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Construct a batch source.
val df = spark
.read
.format("pulsar")
.option("service.url", "pulsar://localhost:6650")
.option("admin.url", "http://localhost:8080")
.option("topicsPattern", "topic.*")
.option("startingOffsets", "earliest")
.option("endingOffsets", "latest")
.load()
df.selectExpr("CAST(__key AS STRING)", "CAST(value AS STRING)")
.as[(String, String)]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment