Skip to content

Instantly share code, notes, and snippets.

@davidhesson
Created September 4, 2018 23:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save davidhesson/c0c77dffe48965650fd5bbb078731900 to your computer and use it in GitHub Desktop.
Save davidhesson/c0c77dffe48965650fd5bbb078731900 to your computer and use it in GitHub Desktop.
Spark Submit - Testing Spark 2.3.1
{
"action": "CreateSubmissionRequest",
"appArgs": ["500"],
"appResource": "s3a://somebucket/examples/spark-examples_2.11-2.3.1.jar",
"clientSparkVersion": "2.3.1",
"mainClass": "org.apache.spark.examples.SparkPi",
"sparkProperties": {
"spark.app.name": "spark_pi_231_ns",
"spark.submit.deployMode": "cluster",
"spark.driver.memory": "6G",
"spark.mesos.driver.labels": "DCOS_SPACE:/spark",
"spark.executor.memory": "6G",
"spark.driver.cores": "2",
"spark.executor.cores": "2",
"spark.cores.max": "38",
"spark.mesos.executor.memory.overHead": "512",
"spark.mesos.driver.constraints": "slave_type:spark",
"spark.driver.supervise": "true",
"spark.master": "mesos://spark-dispatcher.marathon.l4lb.thisdcos.directory:7077"
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment