Skip to content

Instantly share code, notes, and snippets.

@fyyying

fyyying/pyspark_read_dataset_from_pandas.py

Last active June 28, 2020 20:13
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save fyyying/094e8596b7c0274cd00d9afb8b166ff7 to your computer and use it in GitHub Desktop.
Save fyyying/094e8596b7c0274cd00d9afb8b166ff7 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
pyspark_read_dataset_from_pandas.py
# Read data from a pandas dataframe
path = "https://gist.githubusercontent.com/fyyying/4aa5b471860321d7b47fd881898162b7/raw/e8606de9a82e13ca6215b340ce260dad60469cba/titanic_dataset.csv"
# Be careful the object type in pandas can not be understood
# Explicitly change to string type
pd_df = pd.read_csv(path)
df = spark.createDataFrame(pd_df)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment