Skip to content

Instantly share code, notes, and snippets.

@samklr

samklr/spark_joins.scala

Last active December 11, 2018 15:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save samklr/a7aaa4d4c01e7b3e61d5265303b7a5c4 to your computer and use it in GitHub Desktop.
Save samklr/a7aaa4d4c01e7b3e61d5265303b7a5c4 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark_joins.scala
import spark.implicits._
case class Row(id: Int, value: String)
val r1 = Seq(Row(1, "A1"), Row(2, "A2"), Row(3, "A3"), Row(4, "A4")).toDS()
val r2 = Seq(Row(3, "A3"), Row(4, "A4"), Row(4, "A4_1"), Row(5, "A5"), Row(6, "A6")).toDS()
r1.show
r2.show
val joinTypes = Seq("inner", "outer", "full", "full_outer", "left", "left_outer", "right", "right_outer", "left_semi", "left_anti")
joinTypes foreach {joinType =>
println(s"${joinType.toUpperCase()} JOIN")
r1.join(right = r2, usingColumns = Seq("id"), joinType = joinType).orderBy("id").show()
}
https://sql.sh/cours/where/exists
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment