Skip to content

Instantly share code, notes, and snippets.

@yvan

yvan/pandas-pyspark-udf.py

Last active February 22, 2023 23:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yvan/a28ba9b15668225fd94534c05f97cdbb to your computer and use it in GitHub Desktop.
Save yvan/a28ba9b15668225fd94534c05f97cdbb to your computer and use it in GitHub Desktop.
Download ZIP
Raw
pandas-pyspark-udf.py
# pandas
df[col4] = df[col4].apply(lambda m: None if m in [None, float('nan'), np.nan, math.nan] else int(float(m)))
# pyspark
def floatint(x):
return int(float(x))
int_udf = F.udf(lambda m: None if m is None else floatint(m))
df = df.withColumn(col4, F.when(F.col(col4).isNotNull(), int_udf(F.col(col4))).otherwise(None))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment