Skip to content

Instantly share code, notes, and snippets.

@kovid-r

kovid-r/pyspark_cheatsheet_filtering_where.py

Created June 13, 2020 11:37
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kovid-r/6724298aa0c9feb1090c2f2745666b8c to your computer and use it in GitHub Desktop.
Save kovid-r/6724298aa0c9feb1090c2f2745666b8c to your computer and use it in GitHub Desktop.
Download ZIP
Using where to filter PySpark Cheatsheet
Raw
pyspark_cheatsheet_filtering_where.py
# Finding info of Ace Ventura films
df.where(F.lower(F.col('title')).like("%ace%")).show()
# Another way to do this
df.where("title like '%ace%'").show()
# Using where clause in sequence
df.where(df.year != '1998').where(df.avg_ratings >= 6.0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment