Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save karpanGit/7f767210b3c36774af3acfc52a99c5e0 to your computer and use it in GitHub Desktop.
Save karpanGit/7f767210b3c36774af3acfc52a99c5e0 to your computer and use it in GitHub Desktop.
pyspark, reverse the order of a dataframe
# revert the order of a dataframe
from pyspark.sql import SparkSession
import pyspark.sql.functions as f
# create spark session
spark = SparkSession.builder.appName('some name').getOrCreate()
# create a frame
df = spark.createDataFrame([('a',12), ('c',10), ('b',9)], ['col1', 'col2'])
df.show()
# reverse the order of the rows
inv_df = df.withColumn('index', f.monotonically_increasing_id()).orderBy(f.col('index').desc()).drop('index')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment