Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@siakon89

siakon89/pyspark_df_filter.py

Created May 15, 2020 18:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save siakon89/c470fb04187fc672d425d593b3e2b420 to your computer and use it in GitHub Desktop.
Save siakon89/c470fb04187fc672d425d593b3e2b420 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
pyspark_df_filter.py
# Filter your DataFrame and select a column
df.filter(df.column1 > 20).select("column2").show()
# Filter your DataFrame with AND
df.filter((df.column1 > 20) & (df.column2 < 10)).select("column2").show()
# Filter your DataFrame with OR
df.filter((df.column1 > 20) | (df.column2 < 10)).select("column2").show()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment