Skip to content

Instantly share code, notes, and snippets.

@nuria

nuria/spark-shell shothand

Created January 21, 2020 20:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nuria/f7d457eb7598e71edb453444403cfcca to your computer and use it in GitHub Desktop.
Save nuria/f7d457eb7598e71edb453444403cfcca to your computer and use it in GitHub Desktop.
Download ZIP
spark-shell
Raw
spark-shell shothand
spark2-shell --master yarn --executor-memory 8G --executor-cores 4 --driver-memory 16G --conf spark.dynamicAllocation.maxExecutors=64 --conf spark.executor.memoryOverhead=2048 --jars /srv/deployment/analytics/refinery/artifacts/refinery-job.jar,/srv/deployment/analytics/refinery/artifacts/refinery-hive.jar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment