Skip to content

Instantly share code, notes, and snippets.

@connerxyz
Created April 26, 2019 04:55
Show Gist options
  • Save connerxyz/3f44ecb78ef6122f94f77034f17d798a to your computer and use it in GitHub Desktop.
Save connerxyz/3f44ecb78ef6122f94f77034f17d798a to your computer and use it in GitHub Desktop.
Write a Python dict to a single file in HDFS as JSON
# You've got a Python dict in the driver program or shell
x = {"a":1,"b":2}
# You want to write it to a single HDFS file in JSON format
sc.parallelize([x]).toDF().coalesce(1).write.mode('append').json(/hdfs/path/...)
@calebddavis
Copy link

did you import anything to do this?

@connerxyz
Copy link
Author

No.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment