Skip to content

Instantly share code, notes, and snippets.

@ebrard

ebrard/spark-submit-k8s.sh

Created February 11, 2018 15:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ebrard/07e214c430b91ea0a12a8e7d0ab6a826 to your computer and use it in GitHub Desktop.
Save ebrard/07e214c430b91ea0a12a8e7d0ab6a826 to your computer and use it in GitHub Desktop.
Download ZIP
Submist Spark job on k8s cluster
Raw
spark-submit-k8s.sh
bin/spark-submit \
--deploy-mode cluster \
--class org.apache.spark.examples.SparkPi \
--master k8s://https://192.168.99.100:8443 \
--kubernetes-namespace default \
--conf spark.executor.instances=1 \
--conf spark.app.name=spark-pi \
--conf spark.executor.memory=500M \
--conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
--conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.shuffle.service.enabled=true \
--conf spark.kubernetes.shuffle.namespace=default \
--conf spark.kubernetes.shuffle.labels="app=spark-shuffle-service,spark-version=2.2.0" \
--conf spark.kubernetes.shuffle.dir=/tmp/spark-local \
--conf spark.local.dir=/tmp/spark-local \
local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment