Skip to content

Instantly share code, notes, and snippets.

@noahgift

noahgift/databricks_mongo

Created December 28, 2016 21:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save noahgift/cef6af711bad4bac3c091013c042d0d0 to your computer and use it in GitHub Desktop.
Save noahgift/cef6af711bad4bac3c091013c042d0d0 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
databricks_mongo
>
import sys
from pyspark.sql import SQLContext, SparkSession
from pyspark import SparkContext, SparkConf
Command took 0.07 seconds -- by noah.gift@18birdies.com at 12/28/2016, 1:57:21 PM on dev-dp-mongo-etl
>
sparkConf = SparkConf().setMaster("local").setAppName("MongoSparkConnectorTour").set("spark.app.id", "MongoSparkConnectorTour")
​
Command took 0.02 seconds -- by noah.gift@18birdies.com at 12/28/2016, 1:57:21 PM on dev-dp-mongo-etl
>
sqlContext = SQLContext(sc)
Command took 0.07 seconds -- by noah.gift@18birdies.com at 12/28/2016, 1:57:40 PM on dev-dp-mongo-etl
>
df = sqlContext.read.format("com.mongodb.spark.sql.DefaultSource").option("spark.mongodb.input.uri", "mongodb://172.31.30.161:27017/user.UserFollow").load()
java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
Command took 0.28 seconds -- by noah.gift@18birdies.com at 12/28/2016, 1:57:43 PM on dev-dp-mongo-etl
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment