Skip to content

Instantly share code, notes, and snippets.

@hadoopsters

hadoopsters/dstream_to_orc.scala

Created September 1, 2017 19:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hadoopsters/e37c2bb40d1853f7738e44baa927d7b6 to your computer and use it in GitHub Desktop.
Save hadoopsters/e37c2bb40d1853f7738e44baa927d7b6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
dstream_to_orc.scala
val mydstream = ... // these usually come from Spark Streaming apps
// they basically contain a chain of RDDs that you can convert to DFs
mydstream.foreachRDD(rdd => {
hiveContext.createDataFrame(rdd)
.write
.option("orc.compress", "snappy")
.mode(SaveMode.Append)
.orc("/this/is/an/hdfs/directory/too")
})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment