Skip to content

Instantly share code, notes, and snippets.

@voycey

voycey/convenience_fields.py

Created July 19, 2021 02:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save voycey/aee0fee1564594979fdf7d7fc7742e95 to your computer and use it in GitHub Desktop.
Save voycey/aee0fee1564594979fdf7d7fc7742e95 to your computer and use it in GitHub Desktop.
Download ZIP
Convenience fields
Raw
convenience_fields.py
mapped_readings_df = mapped_readings_df \
.withColumn("week_of_year", F.weekofyear(reading_time)) \
.withColumn("date_str", F.regexp_replace(F.col("period_end").substr(1,10), "-", "")) \
.withColumn("day_of_month", F.dayofmonth(reading_time)) \
.withColumn("month", F.month(reading_time)) \
.withColumn("year", F.year(reading_time)) \
.withColumn("hour", F.hour(reading_time)) \
.withColumn("minute", F.minute(reading_time)) \
.withColumn("reading_date_time", reading_time) \
.withColumn("ingestion_date", F.current_timestamp()) \
.drop("reading_time")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment