Skip to content

Instantly share code, notes, and snippets.

@oaustegard

oaustegard/scalding_alice.scala

Forked from johnynek/scalding_alice.scala
Last active August 29, 2015 14:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save oaustegard/ff0bfecc8a672b1faada to your computer and use it in GitHub Desktop.
Save oaustegard/ff0bfecc8a672b1faada to your computer and use it in GitHub Desktop.
Download ZIP
Learn Scalding with Alice
Raw
scalding_alice.scala
/**
Get the last line of text in Alice in Wonderland that contains each of the top 20 frequent words in the book
NOT ACTUALLY TESTED, just logically combined. See fork source for original version that is mulch ore likely to work.
*/
import scala.io.Source
val alice = Source.fromURL("http://www.gutenberg.org/files/11/11.txt").getLines.zipWithIndex.toList
val aliceWords = TypedPipe.from(alice).flatMap { case (text, _) => text.split("\\s+").toList }
val wordCount = aliceWords.map { word => (word, 1L) }.group.sum
val top20 = wordCount.groupAll.sortBy { case (word, count) => -count }.take(20).values
val lastLine = alicePipe.flatMap { case (text, line) => text.split("\\s+").toList.map { word => (word, line) }}.group.max
lastLine.map { case (word, lastLine) => (lastLine, word) }.group.join(alicePipe.swap.group).dump
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment