Skip to content

Instantly share code, notes, and snippets.

@ramonpin
Forked from yaravind/spark-rest-submit.sh
Created April 11, 2018 06:52
Show Gist options
  • Save ramonpin/16f8b28c3554449fc8b7dbcf399e80fd to your computer and use it in GitHub Desktop.
Save ramonpin/16f8b28c3554449fc8b7dbcf399e80fd to your computer and use it in GitHub Desktop.
Submit apps (SparkPi as e.g.) to spark cluster using rest api
curl -X POST -d http://master-host:6066/v1/submissions/create --header "Content-Type:application/json" --data '{
"action": "CreateSubmissionRequest",
"appResource": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
"clientSparkVersion": "2.0.0",
"appArgs": [ "10" ],
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass": "org.apache.spark.examples.SparkPi",
"sparkProperties": {
"spark.jars": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
"spark.driver.supervise":"false",
"spark.executor.memory": "512m",
"spark.driver.memory": "512m",
"spark.submit.deployMode":"cluster",
"spark.app.name": "SparkPi",
"spark.master": "spark://master-host:6066"
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment