Skip to content

Instantly share code, notes, and snippets.

@eduardorost

eduardorost/spark.scala

Created September 1, 2019 02:14
Show Gist options
  • Save eduardorost/d0e467a6ef8954400e1499574ac26938 to your computer and use it in GitHub Desktop.
Save eduardorost/d0e467a6ef8954400e1499574ac26938 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark.scala
val sparkConf: SparkConf = new SparkConf()
.setMaster("local[*]")
.set("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2")
.set("spark.streaming.driver.writeAheadLog.closeFileAfterWrite", "true")
.set("spark.streaming.receiver.writeAheadLog.closeFileAfterWrite", "true")
val spark: SparkSession = SparkSession
.builder()
.config(sparkConf)
.getOrCreate()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment