Skip to content

Instantly share code, notes, and snippets.

@oaustegard
Forked from johnynek/scalding_alice.scala
Last active August 29, 2015 14:27
Show Gist options
  • Save oaustegard/ff0bfecc8a672b1faada to your computer and use it in GitHub Desktop.
Save oaustegard/ff0bfecc8a672b1faada to your computer and use it in GitHub Desktop.
Learn Scalding with Alice
/**
Get the last line of text in Alice in Wonderland that contains each of the top 20 frequent words in the book
NOT ACTUALLY TESTED, just logically combined. See fork source for original version that is mulch ore likely to work.
*/
import scala.io.Source
val alice = Source.fromURL("http://www.gutenberg.org/files/11/11.txt").getLines.zipWithIndex.toList
val aliceWords = TypedPipe.from(alice).flatMap { case (text, _) => text.split("\\s+").toList }
val wordCount = aliceWords.map { word => (word, 1L) }.group.sum
val top20 = wordCount.groupAll.sortBy { case (word, count) => -count }.take(20).values
val lastLine = alicePipe.flatMap { case (text, line) => text.split("\\s+").toList.map { word => (word, line) }}.group.max
lastLine.map { case (word, lastLine) => (lastLine, word) }.group.join(alicePipe.swap.group).dump
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment