Skip to content

Instantly share code, notes, and snippets.

@Kornel

Kornel/run-spark-shell.sh

Created October 6, 2015 09:41
Show Gist options
  • Save Kornel/7c70a9d84e6703d14dc3 to your computer and use it in GitHub Desktop.
Save Kornel/7c70a9d84e6703d14dc3 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
run-spark-shell.sh
#!/bin/sh
SSHELL=~/spark-1.5.0-bin-2.5.0-cdh5.3.1/bin/spark-shell
#SSHELL=~/spark-1.4.0-bin-2.5.0-cdh5.3.1/bin/spark-shell
#SSHELL=spark-shell
SSHELL=spark-shell1.5
JAR=./lib/reco-analyzer-assembly-1.12.1-SNAPSHOT.jar
#SPARK_HOME=~/spark-1.5.0-bin-2.5.0-cdh5.3.1 $SSHELL --master yarn --driver-memory 3g --jars $JAR --queue reco_prod --num-executors 20 --driver-java-options "-Dspark.executor.memory=4g"
$SSHELL --master yarn --driver-memory 3g --jars $JAR --queue reco_prod --num-executors 20 --driver-java-options "-Dspark.executor.memory=4g"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment