Skip to content

Instantly share code, notes, and snippets.

@Cowa

Cowa/ngram-prob-index.scala

Created October 21, 2015 19:58
Show Gist options
  • Save Cowa/dfea4398eda88419cc70 to your computer and use it in GitHub Desktop.
Save Cowa/dfea4398eda88419cc70 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
ngram-prob-index.scala
val sumIndex = ngramWithCount.groupBy { case (k, v) => k.take(n - 1) }.mapValues(_.values.sum)
val ngramWithProbabilityFaster = ngramWithCount.map { case (k, v) =>
(k, v.toDouble / sumIndex(k.take(n - 1)))
}
// Map(List(eating, the) -> 1.0, List(blue, sky) -> 1.0, List(red, koala) -> 1.0, List(near, the) -> 1.0, List(The, blue) -> 1.0, List(the, sky) -> 0.5, List(koala, eating) -> 1.0, List(sky, is) -> 1.0, List(the, red) -> 0.5, List(is, near) -> 1.0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment