Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Learn Scalding with Alice
/**
Get the last line of text in Alice in Wonderland that contains each of the top 20 frequent words in the book
NOT ACTUALLY TESTED, just logically combined. See fork source for original version that is mulch ore likely to work.
*/
import scala.io.Source
val alice = Source.fromURL("http://www.gutenberg.org/files/11/11.txt").getLines.zipWithIndex.toList
val aliceWords = TypedPipe.from(alice).flatMap { case (text, _) => text.split("\\s+").toList }
val wordCount = aliceWords.map { word => (word, 1L) }.group.sum
val top20 = wordCount.groupAll.sortBy { case (word, count) => -count }.take(20).values
val lastLine = alicePipe.flatMap { case (text, line) => text.split("\\s+").toList.map { word => (word, line) }}.group.max
lastLine.map { case (word, lastLine) => (lastLine, word) }.group.join(alicePipe.swap.group).dump
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment