As of Feburary-14-2019, Spark does not have integration with EKS as spark binary would require to use aws-iam-authentator to fetch credential to authenticate to EKS cluster which will be integrated to kubernetes-client 4.1.2 release Ref: fabric8io/kubernetes-client#1358
In order to integrate spark binary with EKS, we will have to do a custom build with fabric 4.1-SHAPSHOT version.
Java8
Apache-maven3.x.x and above
1) git clone https://github.com/fabric8io/kubernetes-client.git
2) cd kubernetes-client/
3) mvn clean install -DskipTests (Maven central repository has fabric8io/kubernetes-client version 4.1.1 which has some issue with EKS aws-iam-authenticator)
4) git clone https://github.com/apache/spark.git
5) cd spark
6) vi resource-managers/kubernetes/core/pom.xml and change kubernetes.client.version from 4.1.0 to 4.1-SNAPSHOT and save
7) Now build a new binary from the above source
./dev/make-distribution.sh --name custom-spark --tgz -Phadoop-2.7 -Pkubernetes
8) This would create a new tgz file spark-3.0.0-SNAPSHOT-bin-custom-spark.tgz
9) Untar: tar -xzvf spark-3.0.0-SNAPSHOT-bin-custom-spark.tgz
10) export MASTER=<EKS server endpoint>
11) Test by running this command: . spark-3.0.0-SNAPSHOT-bin-custom-spark/bin/spark-submit --master k8s://${MASTER} --deploy-mode cluster --class SimpleApp --name SimpleApp --conf spark.executor.instances=16 --conf spark.kubernetes.container.image=jamesrcounts/hello-kubernetes:latest local:///opt/spark/jars/hello-kubernetes_2.11-0.1.jar