Spark in Kubernetes

spark_k8s.md

Build Kubernetes with Spark support

From: https://github.com/apache-spark-on-k8s/spark/blob/branch-2.2-kubernetes/resource-managers/kubernetes/README.md#building-spark-with-kubernetes-support

• git clone https://github.com/apache-spark-on-k8s/spark.git
• build/mvn install -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests
• dev/make-distribution.sh --tgz -Phadoop-2.7 -Pkubernetes

... to have spark-2.2.0-k8s-0.5.0-SNAPSHOT-bin-2.7.3.tgz

• cp spark-2.2.0-k8s-0.5.0-SNAPSHOT-bin-2.7.3.tgz ..
• cd ..
• tar -xcf spark-2.2.0-k8s-0.5.0-SNAPSHOT-bin-2.7.3.tgz
• cd spark-2.2.0-k8s-0.5.0-SNAPSHOT-bin-2.7.3

... and then run the following ...

An alternative instead of building from the sources is to use pre-built docker images (see https://hub.docker.com/u/kubespark/) like this:

bin/spark-submit \
  --deploy-mode cluster \
  --class org.apache.spark.examples.SparkPi \
  --master k8s://https://192.168.99.100:8443 \
  --kubernetes-namespace default \
  --conf spark.executor.instances=2 \
  --conf spark.app.name=spark-pi \
  --conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.4.0 \
  --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.4.0 \
  --conf spark.kubernetes.initcontainer.docker.image=kubespark/spark-init:v2.2.0-kubernetes-0.4.0 \
  local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.4.0.jar

... and it fails due to lack of CPU / resources in the Kubernetes cluster ...

BTW for monitoring pods remember kubectl get pods --watch

TODO

• Try connecting Spring Boot with Slack

Issues

• To prevent DOCKER_API_VERSION issue (1.23 vs 1.24) ended up uninstalling Docker for Mac and installing Docker Toolbox 17.06.2-ce via Homebrew