Calculating the size of a Spark data frame

get_spark_dataframe_size.py

files = [
  "file://path"
]
df = spark.read.json(files)
catalyst_plan = df._jdf.queryExecution().logical()
df_size_read = spark._jsparkSession.sessionState().executePlan(catalyst_plan).optimizedPlan().stats().sizeInBytes()

Thanks for this snippet, but it is not working for me.

any idea?

Hm... @VibhavariBellutagi19 maybe it's not working on your version of Spark. It is some undocumented feature.