Skip to content

Instantly share code, notes, and snippets.

@YordanGeorgiev

YordanGeorgiev/spark-dataframe-fullouter-join-on-nullable-columns.scala

Last active February 17, 2018 10:42
Show Gist options
  • Save YordanGeorgiev/20ccf8feabee452afe4a7395a5b812fd to your computer and use it in GitHub Desktop.
Save YordanGeorgiev/20ccf8feabee452afe4a7395a5b812fd to your computer and use it in GitHub Desktop.
Download ZIP
[full outer join on nullable columns for spark dataframe] how-to apply a full outer join on a spark dataframe #scala #spark #dataframe #joins
Raw
spark-dataframe-fullouter-join-on-nullable-columns.scala
val lstKeyCols = List("col1" , "col2" , "col3" )
dfLeft
.join(
dfRight,
dfLeft("col1") <=> dfRight("col1_")
&& dfLeft("col2") <=> dfRight("col2_")
&& dfLeft("col3") <=> dfRight("col3_"),
"fullouter"
)
.drop(lstKeyCols.map(_ + "_"): _*)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment