Skip to content

Instantly share code, notes, and snippets.

@mjhb

mjhb/df_map.py

Created March 25, 2017 17:53
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mjhb/577f2759528b3ce262db13bdd6d86081 to your computer and use it in GitHub Desktop.
Save mjhb/577f2759528b3ce262db13bdd6d86081 to your computer and use it in GitHub Desktop.
Download ZIP
PySpark DataFrame map example
Raw
df_map.py
df2=df.rdd.map(lambda row: Row(**dict(row.asDict(), test_name=row.TEST_NAME.strip().lower()))).toDF()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment