Skip to content

Instantly share code, notes, and snippets.

@dwerbam
Last active October 8, 2021 06:40
Show Gist options
  • Save dwerbam/0bb83e793841ecdb4f49230b079ca210 to your computer and use it in GitHub Desktop.
Save dwerbam/0bb83e793841ecdb4f49230b079ca210 to your computer and use it in GitHub Desktop.
spark case_when raw implementation
dfletters <- data.frame(letter=letters)
sparkletters <- sdf_copy_to(sc, dfletters, 'sometable', overwrite=T)
# or using sdf_register(name='sometable')
sparkletters
df2 <- sdf_sql(sc, "select letter as A1,
CASE WHEN letter = 'b' OR letter='d'
THEN 'itsBorD'
ELSE
CASE WHEN letter = 'c'
THEN 'itsC'
ELSE 'Other'
END
END AS IS_B_COL
from sometable")
df2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment