Skip to content

Instantly share code, notes, and snippets.

@jbenninghoff
Last active March 26, 2019 19:34
Show Gist options
  • Save jbenninghoff/2c7af1eac47a84b038f1 to your computer and use it in GitHub Desktop.
Save jbenninghoff/2c7af1eac47a84b038f1 to your computer and use it in GitHub Desktop.
Spark TeraSort launch example
#!/bin/bash
/opt/mapr/spark/spark-1.2.1/bin/spark-submit --master yarn-client \
--class org.apache.spark.examples.terasort.TeraGen \
--name 'TeraGen' \
--conf 'mapreduce.terasort.num.partitions=5' \
--executor-cores 30 \
--executor-memory 7G \
--num-executors 9 \
terasort-project_2.10-1.0.jar 50G /user/$USER/spark-terasort
/opt/mapr/spark/spark-1.2.1/bin/spark-submit --master yarn-client \
--class org.apache.spark.examples.terasort.TeraSort \
--name 'TeraSort' \
--conf 'mapreduce.terasort.num.partitions=5' \
--executor-cores 30 \
--executor-memory 7G \
--num-executors 9 \
terasort-project_2.10-1.0.jar /user/$USER/spark-terasort /user/$USER/terasort-output
# --conf 'spark.driver.extraClassPath=/opt/mapr/lib/libprotodefs-4.0.1-mapr.jar:/opt/mapr/lib/protobuf-java-2.5.0.jar:/opt/mapr/lib/guava-13.0.1.jar' \
@varshitpallem
Copy link

hello this is siva iam trying u r command is given iam not get output can u helo me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment