Skip to content

Instantly share code, notes, and snippets.

@hanifa2102

hanifa2102/PySpark CheatSheet

Last active April 18, 2019 06:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hanifa2102/ca9cfd9ac7c8cab1316881156fd24e59 to your computer and use it in GitHub Desktop.
Save hanifa2102/ca9cfd9ac7c8cab1316881156fd24e59 to your computer and use it in GitHub Desktop.
Download ZIP
PySpark Intro
Raw
PySpark CheatSheet
- Create Spark Context to connect to Spark Cluster
- Have a spark session (use Static method to retrieve it)
@hanifa2102
Copy link
Author

  • Modify Columns
  • df = spark.table("flights")
  • df.withColumn("mins",df.hour*60)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment