Skip to content

Instantly share code, notes, and snippets.

@krishnanraman
Created February 8, 2018 02:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save krishnanraman/6a530a72a28bcdba5274f07064fbcd15 to your computer and use it in GitHub Desktop.
Save krishnanraman/6a530a72a28bcdba5274f07064fbcd15 to your computer and use it in GitHub Desktop.
Create Spark DataFrame From List
+---+---+
| x| y|
+---+---+
| 1| 0|
| 2| 0|
| 3| 0|
| 4| 0|
| 5| 0|
| 6| 0|
| 7| 0|
| 8| 0|
| 9| 0|
| 10| 1|
| 11| 1|
| 12| 1|
| 13| 1|
| 14| 1|
| 15| 1|
| 16| 1|
| 17| 1|
| 18| 1|
| 19| 1|
| 20| 2|
| 21| 2|
| 22| 2|
| 23| 2|
| 24| 2|
| 25| 2|
| 26| 2|
| 27| 2|
| 28| 2|
| 29| 2|
| 30| 3|
| 31| 3|
| 32| 3|
| 33| 3|
| 34| 3|
| 35| 3|
| 36| 3|
| 37| 3|
| 38| 3|
| 39| 3|
| 40| 4|
| 41| 4|
| 42| 4|
| 43| 4|
| 44| 4|
| 45| 4|
| 46| 4|
| 47| 4|
| 48| 4|
| 49| 4|
| 50| 5|
| 51| 5|
| 52| 5|
| 53| 5|
| 54| 5|
| 55| 5|
| 56| 5|
| 57| 5|
| 58| 5|
| 59| 5|
| 60| 6|
| 61| 6|
| 62| 6|
| 63| 6|
| 64| 6|
| 65| 6|
| 66| 6|
| 67| 6|
| 68| 6|
| 69| 6|
| 70| 7|
| 71| 7|
| 72| 7|
| 73| 7|
| 74| 7|
| 75| 7|
| 76| 7|
| 77| 7|
| 78| 7|
| 79| 7|
| 80| 8|
| 81| 8|
| 82| 8|
| 83| 8|
| 84| 8|
| 85| 8|
| 86| 8|
| 87| 8|
| 88| 8|
| 89| 8|
| 90| 9|
| 91| 9|
| 92| 9|
| 93| 9|
| 94| 9|
| 95| 9|
| 96| 9|
| 97| 9|
| 98| 9|
| 99| 9|
|100| 10|
+---+---+
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val data = (1 to 100).toList.map{ x=> List(x,x/10) }.map{ x:List[Int] => Row.fromSeq(x)}
val rdd = spark.sparkContext.makeRDD(data)
val cols = List(StructField("x", IntegerType, false), StructField("y", IntegerType,false))
val df = spark.createDataFrame(rdd, StructType(cols))
df.show(100)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment