Skip to content

Instantly share code, notes, and snippets.

@rafaelfelix

rafaelfelix/spark-on-k8s-run-sample-job.sh

Last active November 14, 2019 16:12
Show Gist options
  • Save rafaelfelix/a3971f5c2a2765053a1cc80715c72328 to your computer and use it in GitHub Desktop.
Save rafaelfelix/a3971f5c2a2765053a1cc80715c72328 to your computer and use it in GitHub Desktop.
Download ZIP
spark-on-k8s-run-sample-job.sh
Raw
spark-on-k8s-run-sample-job.sh
#!/bin/bash
IMAGE_REPO=$(kubectl get svc registry-docker-registry -o=jsonpath='{.spec.clusterIP}'):5000
export SPARK_HOME=$(pwd)/spark
$SPARK_HOME/bin/spark-submit \
--master k8s://https://$(minikube ip):8443 \
--conf spark.kubernetes.container.image=$IMAGE_REPO/spark \
--class org.apache.spark.examples.SparkPi \
$SPARK_HOME/examples/jars/spark-examples_2.11-2.4.4.jar
# check the executor pods in another terminal window while running
kubectl get pods -w
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment