Skip to content

Instantly share code, notes, and snippets.

@dgadiraju

dgadiraju/spark-row-level-transformations-flatmap.scala

Created November 23, 2017 11:35
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save dgadiraju/0d4677838c9e7919a43edc9848c9dd8e to your computer and use it in GitHub Desktop.
Save dgadiraju/0d4677838c9e7919a43edc9848c9dd8e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark-row-level-transformations-flatmap.scala
// Row level transformations using flatMap
val l = List("Hello", "How are you doing", "Let us perform word count", "As part of the word count program", "we will see how many times each word repeat")
val l_rdd = sc.parallelize(l)
val l_map = l_rdd.map(ele => ele.split(" "))
val l_flatMap = l_rdd.flatMap(ele => ele.split(" "))
val wordcount = l_flatMap.map(word => (word, "")).countByKey
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment