Skip to content

Instantly share code, notes, and snippets.

Avatar

Elena Savinova falkerl

View GitHub Profile
View gist:f62fd4d4b60eb2a36ed1e9073d9b17db
test("EEA_UK_deaths") {
val df = spark.read.option("header", true)
.csv("/Users/elena/Downloads/Morticd10_part*")
df.createTempView("data")
val code2country = spark.read.option("header", true)
.csv("/Users/elena/Downloads/WHO_country_codes.csv")
@falkerl
falkerl / test.scala
Last active Mar 9, 2021
Vaccine combinations
View test.scala
val df = spark.read.option("header", true)
.csv("/Users/elena/Downloads/vaccine_combinations.csv")
df.createTempView("data")
val diseases = df.columns.filter(_ != "ID")
diseases.map(d => df.where(col(d) === lit(1)).select(col("ID"), lit(d).as("disease")))
.reduce(_ union _)
.createTempView("vac2dis")