Skip to content

Instantly share code, notes, and snippets.

@skonto
Created June 13, 2018 10:44
Show Gist options
  • Save skonto/db2192ab81b3aa36f9c5f67db627eb1c to your computer and use it in GitHub Desktop.
Save skonto/db2192ab81b3aa36f9c5f67db627eb1c to your computer and use it in GitHub Desktop.
Dispatcher passes the following command to mesos:
18/06/13 10:30:45 INFO MesosClusterScheduler: Command for the driver (test):./bin/spark-submit --name test --master mesos://zk://master.mesos:2181/mesos --driver-cores 1.0 --driver-memory 1000M --class org.apache.spark.examples.SparkPi --conf spark.executor.extraJavaOptions="-Dexecutor.test.param1=\"value 1\" -Dexecutor.test.param2=value\\ 2 -Dexecutor.test.path=\$PATH" --conf spark.mesos.containerizer=mesos --conf spark.driver.supervise=false --conf spark.app.name=test --conf spark.driver.memory=1000M --conf spark.mesos.executor.docker.image=skonto/spark-test:test --conf spark.driver.cores=1.0 --conf spark.driver.extraJavaOptions="-Dspark.test.param1=\"value 1\" -Dspark.test.param2=value\\ 2 -Dspark.test.path=\$PATH" $MESOS_SANDBOX/spark-examples_2.11-2.3.0.jar 10000
spark-class will run this at the executor side:
exec /usr/lib/jvm/jre1.8.0_152/bin/java -cp '/opt/spark/dist/conf/:/opt/spark/dist/jars/*:/etc/hadoop/' '-Dexecutor.test.param1=value 1' '-Dexecutor.test.param2=value 2' '-Dexecutor.test.path=$PATH' -Xmx1024m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@ip-10-0-0-77.us-west-1.compute.internal:33887 --executor-id 9 --hostname 10.0.0.77 --cores 3 --app-id bddf96ce-45f6-45f7-ac83-5a3622cafc41-0032-driver-20180613103044-0001
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment