Create Spark DataFrame From List[Any]

gistfile1.scala

   // Spark 2.1
    val spark = SparkSession.builder().master("local").getOrCreate()

    // Given a list of mixture of strings in integers
    val values = List("20030100013280", 1.0)

    // Create `Row` from `Seq`
    val row = Row.fromSeq(values)

    // Create `RDD` from `Row`
    val rdd = spark.sparkContext.makeRDD(List(row))

    // Create schema fields
    val fields = List(
      StructField("First Column", StringType, nullable = false),
      StructField("Second Column", DoubleType, nullable = false)
    )

    // Create `DataFrame`
    val dataFrame = spark.createDataFrame(rdd, StructType(fields))

    // Done! Yay!
    dataFrame.show(1)

    +--------------+-------------+
    |  First Column|Second Column|
    +--------------+-------------+
    |20030100013280|          1.0|
    +--------------+-------------+

Had to import:

import org.apache.spark.sql._
import org.apache.spark.sql.types._

Thanks for this. Worked perfectly.

Thanks for the code snippet. Helped a lot.

@yzhong52 If I wanted to save millions of rows, what is the best way to extend this?

@ChinmaySKulkarni suggest reading this:

https://spark.apache.org/docs/latest/sql-data-sources-parquet.html