Skip to content

Instantly share code, notes, and snippets.

@mohak1712

mohak1712/esdata.scala

Created August 13, 2022 08:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mohak1712/d480263c2b6d0dd5d76deedc8c865922 to your computer and use it in GitHub Desktop.
Save mohak1712/d480263c2b6d0dd5d76deedc8c865922 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
esdata.scala
getRedshiftDBData(spark, bmReturns).withColumn("benchmark_vs_fund_returns",
 from_json(col("benchmark_vs_fund_returns"), MapType(StringType,
 ArrayType(DoubleType)))).coalesce(1)
 .write
 .option("es.nodes.wan.only", "true")
 .option("es.mapping.id", "id")
 .format("org.elasticsearch.spark.sql")
 .option("es.write.operation", "upsert")
 .mode(SaveMode.Append)
 .save("fund-details")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment