Skip to content

Instantly share code, notes, and snippets.

@hadoopsters

hadoopsters/rdd_to_orc.scala

Last active September 1, 2017 19:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hadoopsters/d07e813f6907b0aa35391ccceee7351c to your computer and use it in GitHub Desktop.
Save hadoopsters/d07e813f6907b0aa35391ccceee7351c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
rdd_to_orc.scala
// import this guy
import org.apache.spark.sql.hive.HiveContext
// this should look familiar
val conf = new SparkConf()
val sc = new SparkContext(conf)
// setup this fella
val hiveContext = new HiveContext(sc)
val rdd = ... // this is where the data goessssss
// boom shocka-locka, turn that RDD into a DF
hiveContext.createDataFrame(rdd)
.option("orc.compress", "snappy")
.mode(SaveMode.Append)
.orc("/this/is/yet/another/hdfs/directory/")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment