Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Starting Query
# this is a custom processor class that we create to handle real-time inference
# we'll show the skeleton code for it below
processor = RealTimeInferenceProcessor()
query = df.writeStream \
.option("checkpointLocation", "dbfs://pathToYourCheckpoint") \ # configure checkpointing in case of job failure
.foreachBatch(processor.process_batch) \ # for each micro-batch, apply this method
.outputMode("append") \
.start() # start the stream query
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment