A "Best of the Best Practices" (BOBP) guide to developing in Python.
- "Build tools for others that you want to be built for you." - Kenneth Reitz
- "Simplicity is alway better than functionality." - Pieter Hintjens
dimasmamot
rvilla87
/ MongoDBsparkConnector.scala
Created
August 6, 2017 16:28
Inserting documents in MongoDB with Spark Connector (Dataframe vs Spark Structured Streaming)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // Dataframe (supported) - read 1 file, no streaming | |
| // Step 1, create the Dataframe source | |
| val fileDF = spark | |
| .read // No streaming | |
| .csv("file/file1.csv") | |
| .selectExpr("CAST(key as String)", // more code with other casting... | |
| ) | |
| // Out [1]: fileDF: org.apache.spark.sql.package.DataFrame = [key: string, country: string ... 6 more fields] | |
| // Step 2, insert Dataframe into MongoDB |
sloria
/ bobp-python.md
Last active
May 1, 2024 08:37
A "Best of the Best Practices" (BOBP) guide to developing in Python.
digitaljhelms
/ gist:4287848
Last active
April 26, 2024 10:44
Git/GitHub branching standards & conventions