Skip to content

Instantly share code, notes, and snippets.

@TomLous
Created April 2, 2017 13:10
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save TomLous/2a50e6504a9f168474f24376dd66b6cc to your computer and use it in GitHub Desktop.
Save TomLous/2a50e6504a9f168474f24376dd66b6cc to your computer and use it in GitHub Desktop.
val stringify = udf((vs: Seq[String]) => vs.mkString(";"))
groupedKvKRecords
.map(
r => r.kvkRecords
.foldRight((0, 0, List.empty[String]))(
(kvkrecord, comb) => (
comb._1 + 1,
kvkrecord.wptf + comb._2,
kvkrecord.dossierNummer :: comb._3
)
)
)
.toDF("num", "wfte", "dossiernummersArray")
.select(
'num,
'wfte,
stringify('dossiernummersArray).as("dossiernummers")
)
.repartition(1)
.write.mode(SaveMode.Overwrite).option("header", true).option("sep", ";").option("quote", "").option("nullValue", "").option("mode", "FAILFAST")
.csv(Config.targetDataFile)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment