working on spark DF partitions. the following creates a Map of columns (key: column name, value: list of values) from the Rows supplied by the DF in each partition

spark_df_rows_into_cols

val names = df.columns.toList
println(names)
df.foreachPartition(rows => {
  var cols = scala.collection.mutable.Map[String, List[Any]]()
  names.foreach(col => cols(col) = List())
  rows.foreach(row => names.zip(row.toSeq).map(x => {
    cols(x._1) = cols(x._1) :+ x._2 
  }))
  println(cols)
})