Skip to content

Instantly share code, notes, and snippets.

View thomasheslinatmatches's full-sized avatar

thomasheslinatmatches

View GitHub Profile
@fahadsiddiqui
fahadsiddiqui / flatten_df.scala
Last active May 22, 2023 19:42
Flatten a nested JSON Spark DataFrame using Scala, Spark 2.2.+ — a custom solution.
def flattenDataFrame(df: DataFrame): DataFrame = {
val fields = df.schema.fields
val fieldNames = fields.map(x => x.name)
for (i <- fields.indices) {
val field = fields(i)
val fieldType = field.dataType
val fieldName = field.name
fieldType match {
case _: ArrayType =>