Skip to content

Instantly share code, notes, and snippets.

@scalactic
Created July 8, 2021 13:19
Show Gist options
  • Save scalactic/f70a47c83f03acc78a3b43610422d45d to your computer and use it in GitHub Desktop.
Save scalactic/f70a47c83f03acc78a3b43610422d45d to your computer and use it in GitHub Desktop.
Create Spark DataFrame with given schema.
import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
val schema = StructType(
Array(
StructField("key", StringType, nullable = false),
StructField("time", IntegerType, nullable = false)
)
)
val data = Seq(
Row("asd", 123123),
Row("asd", 123123)
)
spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment