Skip to content

Instantly share code, notes, and snippets.

@yvan

yvan/pandas-pyspark-regex-clean.py

Last active February 22, 2023 23:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yvan/d7b62b43c7a0df380d0e92d45c249f0b to your computer and use it in GitHub Desktop.
Save yvan/d7b62b43c7a0df380d0e92d45c249f0b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
pandas-pyspark-regex-clean.py
# pandas
df[col3] = df[col3].replace(regex=r"[^0-9\\.]", value="")
df[col3] = pd.to_numeric(df[col3], errors='coerce')
# pyspark
df = df.withColumn(col3, F.regexp_replace(F.col(col3), r"[^0-9\\.]", '').cast("double"))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment