Skip to content

Instantly share code, notes, and snippets.

@conorfennell
Created January 24, 2016 12:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save conorfennell/8fc55a085d0e0345ee67 to your computer and use it in GitHub Desktop.
Save conorfennell/8fc55a085d0e0345ee67 to your computer and use it in GitHub Desktop.
inferring schema with dataframes
val sc: SparkContext // An existing SparkContext.
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
// Create the DataFrame
val df = sqlContext.read.json("examples/src/main/resources/people.json")
// Show the content of the DataFrame
df.show()
// age name
// null Michael
// 30 Andy
// 19 Justin
// Print the schema in a tree format
df.printSchema()
// root
// |-- age: long (nullable = true)// |-- name: string (nullable = false)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment