Skip to content

Instantly share code, notes, and snippets.

@cerisier
Last active April 10, 2016 14:28
Show Gist options
  • Save cerisier/1e3a7d5fca8596ee054a0428470c1f89 to your computer and use it in GitHub Desktop.
Save cerisier/1e3a7d5fca8596ee054a0428470c1f89 to your computer and use it in GitHub Desktop.
/**
* Pass each value in the key-value pair RDD through a map function without changing the keys;
* this also retains the original RDD's partitioning.
*/
def mapValues[U](f: V => U): RDD[(K, U)] = self.withScope {
val cleanF = self.context.clean(f)
new MapPartitionsRDD[(K, U), (K, V)](self,
(context, pid, iter) => iter.map { case (k, v) => (k, cleanF(v)) },
preservesPartitioning = true)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment