Skip to content

Instantly share code, notes, and snippets.

@canimus

canimus/pyspark_timestamp_cast.py

Last active May 11, 2021 12:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save canimus/d44d8c73754f90d92e4025104e47fe95 to your computer and use it in GitHub Desktop.
Save canimus/d44d8c73754f90d92e4025104e47fe95 to your computer and use it in GitHub Desktop.
Download ZIP
Pyspark Cast 100 columns
Raw
pyspark_timestamp_cast.py
from functools import reduce
from operator import methodcaller
import pyspark.sql.functions as F
_ts = lambda dataFrame, col: methodcaller('withColumn', f'{col}TimeStamp', F.to_timestamp(F.col(col)/1000))(dataFrame)
reduce(lambda a,b: _ts(a,b), ['start','stop'], df).select('startTimeStamp', 'stopTimeStamp').show(truncate=False)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment