Skip to content

Instantly share code, notes, and snippets.

@spektom

spektom/sql-wrap.scala

Created August 20, 2015 14:07
Show Gist options
  • Save spektom/99f7330154be6c47e637 to your computer and use it in GitHub Desktop.
Save spektom/99f7330154be6c47e637 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
sql-wrap.scala
val q = new org.apache.spark.sql.SQLContext(sc);
q.load("parquet", Map("path" -> "s3://raw-data/...", "mergeSchema" -> "false"))
.registerTempTable("organic")
q.sql("SELECT * FROM organic ...")
.map(_.mkString("\t"))
.coalesce(1, true)
.saveAsTextFile("s3://results-bucket/...")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment