Skip to content

Instantly share code, notes, and snippets.

@erkkel
Created April 30, 2018 18:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed to your computer and use it in GitHub Desktop.
Save erkkel/c04a0b5ca60ad755cf62e9ad18e5b7ed to your computer and use it in GitHub Desktop.
[Error] Deploying Spark to Google Kubernetes Engine
spark-kubernetes-driver Apr 29, 2018, 3:49:34 AM at org.apache.spark.deploy.rest.k8s.ResourceStagingServer.main(ResourceStagingServer.scala)
spark-kubernetes-driver Apr 29, 2018, 3:49:34 AM at org.apache.spark.deploy.rest.k8s.ResourceStagingServer$.main(ResourceStagingServer.scala:99)
spark-kubernetes-driver Apr 29, 2018, 3:49:34 AM at org.apache.spark.deploy.rest.k8s.SparkConfPropertiesParser$.getSparkConfFromPropertiesFile(SparkConfPropertiesParser.scala:34)
spark-kubernetes-driver Apr 29, 2018, 3:49:34 AM Exception in thread "main" java.lang.IllegalArgumentException: Server properties file given at /opt/spark/work-dir/driver does not exist or is not a file.
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM + /sbin/tini -s -- /opt/spark/bin/spark-class org.apache.spark.deploy.rest.k8s.ResourceStagingServer driver
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM + '[' -z root:x:0:0:root:/root:/bin/ash ']'
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM + uidentry=root:x:0:0:root:/root:/bin/ash
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM ++ getent passwd 0
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM + mygid=0
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM ++ id -g
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM + myuid=0
spark-kubernetes-driver Apr 29, 2018, 3:49:33 AM ++ id -u
@erkkel
Copy link
Author

erkkel commented Apr 30, 2018

This is the command used that created this error

spark-submit \
  --deploy-mode cluster \
  --class org.apache.spark.examples.SparkPi \
  --master k8s://https://35.225.152.61:443 \
  --conf spark.executor.instances=2 \
  --conf spark.app.name=spark-pi \
  --conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.2.0-kubernetes-0.5.0 \
  --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.2.0-kubernetes-0.5.0 \
  --conf spark.kubernetes.container.image=erkkel/eric:spark \
  local:///opt/spark/examples/jars/spark-examples_2.11-2.2.0-k8s-0.5.0.jar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment