Skip to content

Instantly share code, notes, and snippets.

@mvillarrealb

mvillarrealb/spark-worker:2.3.1.Dockerfile

Last active September 23, 2018 17:26
Show Gist options
  • Save mvillarrealb/70c88540c50a210ca0e09c9d99fe922e to your computer and use it in GitHub Desktop.
Save mvillarrealb/70c88540c50a210ca0e09c9d99fe922e to your computer and use it in GitHub Desktop.
Download ZIP
A custom docker image based on spark-base:2.3.1 image, used to create a spark worker containers
Raw
spark-worker:2.3.1.Dockerfile
FROM spark-base:2.3.1
COPY start-worker.sh /
ENV SPARK_WORKER_WEBUI_PORT 8081
ENV SPARK_WORKER_LOG /spark/logs
ENV SPARK_MASTER "spark://spark-master:7077"
EXPOSE 8081
CMD ["/bin/bash", "/start-worker.sh"]
Raw
start-worker.sh
#!/bin/bash
. "/spark/sbin/spark-config.sh"
. "/spark/bin/load-spark-env.sh"
mkdir -p $SPARK_WORKER_LOG
export SPARK_HOME=/spark
ln -sf /dev/stdout $SPARK_WORKER_LOG/spark-worker.out
/spark/sbin/../bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port $SPARK_WORKER_WEBUI_PORT $SPARK_MASTER >> $SPARK_WORKER_LOG/spark-worker.out
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment