Skip to content

Instantly share code, notes, and snippets.

@oneryalcin
Last active September 23, 2019 20:58
Show Gist options
  • Save oneryalcin/30e248cc87d7a0b7d886f2d1dc5b329b to your computer and use it in GitHub Desktop.
Save oneryalcin/30e248cc87d7a0b7d886f2d1dc5b329b to your computer and use it in GitHub Desktop.
2 Sparkify create spark context
# create a Spark session, if not there is or get an existing one
spark = SparkSession \
.builder \
.appName("Sparkify The music streaming platform churn detection") \
.getOrCreate()
# Check the current Spark Config
spark.sparkContext.getConf().getAll()
>> [('spark.app.id', 'local-1569248217329'),
('spark.driver.port', '46135'),
('spark.rdd.compress', 'True'),
('spark.driver.host', 'cdfb8bf7d5b2'),
('spark.serializer.objectStreamReset', '100'),
('spark.app.name', 'Sparkify The music streaming platform churn detection'),
('spark.master', 'local[*]'),
('spark.executor.id', 'driver'),
('spark.submit.deployMode', 'client'),
('spark.ui.showConsoleProgress', 'true')]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment