Skip to content

Instantly share code, notes, and snippets.

@TomLous TomLous/KvKDataset.scala
Last active Apr 25, 2017

Embed
What would you like to do?
import spark.implicits._
val kvKDataset = spark.read.option("header", true).option("sep", ";").option("ignoreLeadingWhiteSpace", true).option("ignoreTrailingWhiteSpace", true).option("quote", """"""").option("nullValue", "").option("mode", "FAILFAST").csv(path)
kvKDataset.select(
'DOSSIERNR.as("dossierNummer"),
'VG_NR.as("vgNummer"),
'HANDELSN30.as("naamShort"),
'HANDN1_30.as("naamShortP1"),
'HANDN2_30.as("naamShortP2"),
'HANDELSN45.as("naamLong"),
'ADRES_VA.as("adresV"),
'PCPLAATSVA.as("postcodePlaatsV"),
'ADRES_CA.as("adresC"),
'PCPLAATSCA.as("postcodePlaatsC"),
'WPFT.as("wptf").cast(IntegerType),
'SBI.as("sbi").cast(IntegerType)
)
.as[KvKRecord]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.