Skip to content

Instantly share code, notes, and snippets.

@harschware
Created May 15, 2018 22:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save harschware/c8c1a2a0e7f4fc5753d006ad8000e7bc to your computer and use it in GitHub Desktop.
Save harschware/c8c1a2a0e7f4fc5753d006ad8000e7bc to your computer and use it in GitHub Desktop.
livy result shows dataframe.schema contains data types
{
"code": "import org.apache.spark.sql._\nvar d = sqlContext.sql(\"SELECT tbl10.`venueid`, tbl10.`venuename`, tbl10.`venuecity`, tbl10.`venuestate`, tbl10.`venueseats`, tbl10.`processing_dttm` AS `venues_processing_dttm` FROM `concerts`.`venues` tbl10\")\nd.schema\n",
"id": 5,
"output": {
"data": {
"text/plain": "import org.apache.spark.sql._\nd: org.apache.spark.sql.DataFrame = [venueid: int, venuename: string, venuecity: string, venuestate: string, venueseats: int, venues_processing_dttm: string]\nres2: org.apache.spark.sql.types.StructType = StructType(StructField(venueid,IntegerType,true), StructField(venuename,StringType,true), StructField(venuecity,StringType,true), StructField(venuestate,StringType,true), StructField(venueseats,IntegerType,true), StructField(venues_processing_dttm,StringType,true))\n"
},
"execution_count": 5,
"status": "ok"
},
"progress": 1.0,
"state": "available"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment