Skip to content

Instantly share code, notes, and snippets.

@ppillay

ppillay/4-using-a-rdd.scala Secret

Created June 16, 2017 04:24
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ppillay/0c95d5b3188efee14dac4189cc8791a1 to your computer and use it in GitHub Desktop.
Save ppillay/0c95d5b3188efee14dac4189cc8791a1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
4-using-a-rdd.scala
val rDD = df.as[FlightDetails].rdd
val resultRDD= rDD.filter(x => x.delay > 0)
.map(x => ((x.origin, x.dest), (x.delay, 1)))
.reduceByKey((a, b) => (a._1 + b._1, a._2 + b._2))
.mapValues(z => z._1/z._2)
.sortBy(_._2, ascending = false)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment