Skip to content

Instantly share code, notes, and snippets.

@bepcyc

bepcyc/spark_extract_json_field_schema.scala

Created June 19, 2020 15:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bepcyc/fa3ee774a030f77811e4c6a39af84354 to your computer and use it in GitHub Desktop.
Save bepcyc/fa3ee774a030f77811e4c6a39af84354 to your computer and use it in GitHub Desktop.
Download ZIP
Get JSON field schema in Spark
Raw
spark_extract_json_field_schema.scala
val path = "s3://some/dir"
val df = spark.read.parquet(path)
val df2 = df.select($"value") // suppose value is a string with JSON
val ds = df2.as[String]
val dsj = spark.read.json(ds)
val schema = dsj.schema // here is your schema
println(schema.json)
println(schema.toDDL)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment