Skip to content

Instantly share code, notes, and snippets.

@ychennay

ychennay/start_query.py

Created August 5, 2021 19:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ychennay/19d54b79066cf8a8a34de07f803db98d to your computer and use it in GitHub Desktop.
Save ychennay/19d54b79066cf8a8a34de07f803db98d to your computer and use it in GitHub Desktop.
Download ZIP
Starting Query
Raw
start_query.py
# this is a custom processor class that we create to handle real-time inference
# we'll show the skeleton code for it below
processor = RealTimeInferenceProcessor()
query = df.writeStream \
.option("checkpointLocation", "dbfs://pathToYourCheckpoint") \ # configure checkpointing in case of job failure
.foreachBatch(processor.process_batch) \ # for each micro-batch, apply this method
.outputMode("append") \
.start() # start the stream query
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment