Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
val comparableDataset = kvKDataset.as("l")
.joinWith(
kvKDataset.as("r"),
$"l.adresV" === $"r.adresV" && $"l.postcodePlaatsV" === $"r.postcodePlaatsV" && $"l.dossierNummer" =!= $"r.dossierNummer"
).map {
case (left, right) => (left, right, Vectors.dense(left.distance(right).toArray))
}
.toDF("left", "right", "features")
.as[(KvKRecord, KvKRecord, Vector)]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.