Skip to content

Instantly share code, notes, and snippets.

@venuktan

venuktan/spark_local_mac

Created December 20, 2018 22:30
Show Gist options
  • Save venuktan/bc4454cd46cc80ed1045a0940948ca9a to your computer and use it in GitHub Desktop.
Save venuktan/bc4454cd46cc80ed1045a0940948ca9a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark_local_mac
import os
os.environ["PYTHONPATH"]="/Users/vtangirala/spark-2.4.0/python/lib/py4j-0.10.7-src.zip:"+os.environ["PYTHONPATH"]
os.environ["PYTHONPATH"]
from pyspark import SparkConf
from pyspark import SparkContext
conf = SparkConf()
conf.setMaster("spark://vtangirala-mac.adaptiveplanning.com:7077").setAppName("venu")
conf.set("spark.cores.max", "1")
conf.set("spark.executor.memory", "1G")
# conf.set("spark.submit.deployMode", "cluster")
sc = SparkContext(conf = conf)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment