Skip to content

Instantly share code, notes, and snippets.

@Sathiyarajan

Sathiyarajan/SparkSQL with json example

Forked from tmcgrath/SparkSQL with json example
Created February 2, 2018 10:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Sathiyarajan/867062da3b28a30bf172fb7775a4d37c to your computer and use it in GitHub Desktop.
Save Sathiyarajan/867062da3b28a30bf172fb7775a4d37c to your computer and use it in GitHub Desktop.
Download ZIP
SparkSQL with json file example
Raw
SparkSQL with json example
// download sample json
http://bit.ly/2gY39Ay
// start spark-shell in same directory as where customers.json
// is downloaded
val customers = sqlContext.jsonFile("customers.json")
// register a temp table
customers.registerTempTable("customers")
// create dataframe with sql
val firstCityState = sqlContext.sql("SELECT first_name, address.city, address.state FROM customers")
firstCityState.show
firstCityState.collect.foreach(println)
// etc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment