Skip to content

Instantly share code, notes, and snippets.

@jose-goncabel

jose-goncabel/PySpark Row Modification via Dicts

Last active March 29, 2019 15:32
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jose-goncabel/06ecc73826006d55d253d692a90a6c91 to your computer and use it in GitHub Desktop.
Save jose-goncabel/06ecc73826006d55d253d692a90a6c91 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
PySpark Row Modification via Dicts
from pyspark.sql import Row
my_df_schema = my_df.schema
def replace_content(a_row):
a_row_dict = a_row.asDict()
# Modify the contents of the dict
a_row_dict["key"] = "new value"
return Row(**a_row_dict)
my_rdd = my_df.rdd.map(lambda k: (k[0], replace_content(k[1]))
my_df = sqlContext.createDataFrame(my_rdd, my_df_schema)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment