Skip to content

Instantly share code, notes, and snippets.

@adekunleba
Created March 23, 2019 18:28
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save adekunleba/8f37c5d40c0da32ff71f388ad909b4c5 to your computer and use it in GitHub Desktop.
Save adekunleba/8f37c5d40c0da32ff71f388ad909b4c5 to your computer and use it in GitHub Desktop.
val conf = new SparkConf().setMaster("local[*").setAppName("simple-app")
val sparkContext = new SparkContext(conf)
//Loading data with Spark Context returns an RDD
val rdd: RDD[String] = sparkContext.textFile("textfile.csv")
//Also you can create an RDD by parallizing an existing Data
val data: Array[Int] = Array(1, 2, 3, 4, 5, 6, 6, 7, 7)
val dataRdd: RDD[Int] = sparkContext.parallelize(data)
//Given a sparkSession Configuration, it's spark context can also be used for the operations above
val spark: SparkSession = SparkSession.builder()
.master("local[*]")
.appName("simple-app")
.getOrCreate()
val sparkContextSpark :SparkContext = spark.sparkContext
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment