Skip to content

Instantly share code, notes, and snippets.

@eduardorost

eduardorost/spark.send.s3.scala

Last active September 1, 2019 01:16
Show Gist options
  • Save eduardorost/c8b2ff3d2226f88a9a1e2fa977ac9d25 to your computer and use it in GitHub Desktop.
Save eduardorost/c8b2ff3d2226f88a9a1e2fa977ac9d25 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark.send.s3.scala
def sendS3(rdd: RDD[String]) = {
spark.read.json(rdd)
.coalesce(1)
.write
.mode("append")
.option("compression", "snappy")
.parquet("s3://bucket/folder")
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment