Skip to content

Instantly share code, notes, and snippets.

View jamesmunro's full-sized avatar

James Munro jamesmunro

34 followers · 17 following
  • London, UK
View GitHub Profile
@jamesmunro
jamesmunro / PySpark DataFrame from many small pandas DataFrames.ipynb
Created October 2, 2023 16:42 — forked from linar-jether/PySpark DataFrame from many small pandas DataFrames.ipynb
Convert a RDD of pandas DataFrames to a single Spark DataFrame using Arrow and without collecting all data in the driver.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.