Skip to content

Instantly share code, notes, and snippets.

@priyanlc

priyanlc/csv_to_parquet.py

Last active April 30, 2020 20:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save priyanlc/0a60d35cc752c601703bd8f2d0f60441 to your computer and use it in GitHub Desktop.
Save priyanlc/0a60d35cc752c601703bd8f2d0f60441 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
csv_to_parquet.py
df=spark.read.option("header","true").option("inferSchema","true) \
.csv(AWS_BUCKET_NAME +"/taxi_fare")
df.write.parquet(AWS_BUCKET_NAME +"/taxi_fare_parquet")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment