Skip to content

Instantly share code, notes, and snippets.

@seregasheypak
Created August 31, 2015 09:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save seregasheypak/fc89a5bec239772614b8 to your computer and use it in GitHub Desktop.
Save seregasheypak/fc89a5bec239772614b8 to your computer and use it in GitHub Desktop.
group and iterate over group values
def groupAndGenerateNewSurrogateKey: Pipe = {
pipe.groupBy('naturalKey){ group =>
group.mapStream[Long, (Long, Long)]('someValueField -> ('someValueField, 'newSurrogateKey)) { items: Iterator[Long] =>
val newSurrogateKey = KeyGen.generate()
println(s"new group key:[$newSurrogateKey]") //outputs generated key
println(s"items: ${items.toList}") //correctly outputs grouped items
items.map((_,newSurrogateKey)).toList
}
}.project('someValueField, 'newSurrogateKey)
}
//returns NOTHING... Why?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment