Skip to content

Instantly share code, notes, and snippets.

@mitliagkas

mitliagkas/start-spark.sh

Created September 15, 2014 18:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mitliagkas/ae64b1f7213a2976504a to your computer and use it in GitHub Desktop.
Save mitliagkas/ae64b1f7213a2976504a to your computer and use it in GitHub Desktop.
Download ZIP
PySpark setup script
Raw
start-spark.sh
#!/bin/sh
export IPYTHON_OPTS="notebook --profile=yannis_spark"
unset SPARK_JAVA_OPTS
export SPARK_DAEMON_JAVA_OPTS="\
-Dspark.default.parallelism=128 \
-Dspark.cores.max=32 \
-Dspark.worker.timeout=500 \
-Dspark.akka.timeout=500 \
"
#-Dspark.executor.memory=668m \
#-Dspark.worker.memory=10g \
export SPARK_DRIVER_MEMORY=20g
#export SPARK_WORKER_MEMORY=1g
#export MASTER=local[16]
nice -n 20 ./spark/bin/pyspark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment