Skip to content

Instantly share code, notes, and snippets.

@daschl
Created October 27, 2016 05:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save daschl/9ca785972b95604b0ffd5fef7623d608 to your computer and use it in GitHub Desktop.
Save daschl/9ca785972b95604b0ffd5fef7623d608 to your computer and use it in GitHub Desktop.
val spark = SparkSession
.builder
.master("local[*]")
.appName("StructuredWordCount")
.getOrCreate()
import spark.implicits._
val lines = spark.readStream
.format("socket")
.option("host", "localhost")
.option("port", 5050)
.load()
val words = lines.as[String].flatMap(_.split(" "))
val wordCounts = words.groupBy("value").count()
val query = wordCounts.writeStream
.outputMode("complete")
.option("checkpointLocation", "mycheckpointlocation")
.option("idField", "value")
.format("com.couchbase.spark.sql")
.start()
query.awaitTermination()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment