Skip to content

Instantly share code, notes, and snippets.

@1ambda

1ambda/df-partition-overwrite.py

Created December 27, 2021 00:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save 1ambda/c0e83f9f65af78859ceb6b3dc25bdb9a to your computer and use it in GitHub Desktop.
Save 1ambda/c0e83f9f65af78859ceb6b3dc25bdb9a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
df-partition-overwrite.py
dfListingCalendarFiltered\
.where(col("dt") == lit("20211101"))\
.repartition(1)\
.write\
.mode("overwrite")\
.format("parquet")\
.partitionBy("dt")\
.option("path", "s3://airbnb-data-lake/db/listing_calendar")\
.saveAsTable("airbnb_db.listing_calendar")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment