Skip to content

Instantly share code, notes, and snippets.

@lukleh

lukleh/spark_code.py

Last active July 29, 2019 16:23
Show Gist options
  • Save lukleh/dc861417f89b2255bf97be8606f2a430 to your computer and use it in GitHub Desktop.
Save lukleh/dc861417f89b2255bf97be8606f2a430 to your computer and use it in GitHub Desktop.
Download ZIP
spark use shuffle service
Raw
spark_code.py
df = spark.sql('select abs(n1 - n2) as rdiff, count(1) as cnt from tempsampledata group by rdiff order by cnt desc')
df.persist()
df.show(n=100, truncate=False)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment