Skip to content

Instantly share code, notes, and snippets.

@bradkarels

bradkarels/spark-local-environment-vars

Created December 3, 2014 18:49
Show Gist options
  • Save bradkarels/248727d49ce2191ebb87 to your computer and use it in GitHub Desktop.
Save bradkarels/248727d49ce2191ebb87 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark-local-environment-vars
# Spark local environment variables
export SPARK_HOME=/home/bkarels/spark/current
export SPARK_MASTER_IP=127.0.0.1
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=8080
#SPARK_MASTER-OPTS=
export SPARK_LOCAL_DIRS=$SPARK_HOME/work
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=4G
#export SPARK_WORKER_WEBUI_PORT=8081
export SPARK_WORKER_INSTANCES=1
export SPARK_DAEMON_MEMORY=768m
@bradkarels
Copy link
Author

One example of how I set environment variables to run a small, local spark cluster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment