Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
(let [..
schema (trans/extract-dataframe-schema (record-builder nil))
..
rdd (spark/map record-builder some-rdd-we-have)
rows (spark/map trans/as-rows rdd)
dataframe (spark/create-data-frame sql-context rows schema)
]
(spark/save-parquert dataframe output-path :overwrite))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment