Skip to content

Instantly share code, notes, and snippets.

@itayB

itayB/spark.Dockerfile

Created March 7, 2021 17:15
Show Gist options
  • Save itayB/061e2aa73eab16ef35bbf04f5ec69192 to your computer and use it in GitHub Desktop.
Save itayB/061e2aa73eab16ef35bbf04f5ec69192 to your computer and use it in GitHub Desktop.
Download ZIP
Spark with AWS support
Raw
spark.Dockerfile
FROM itayb/spark:3.1.1-hadoop-3.2.0
RUN apt-get update && apt install -y \
wget \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /var/cache/apt/*
RUN cd /opt/spark/jars \
&& wget "https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.888/aws-java-sdk-bundle-1.11.888.jar"
RUN cd /opt/spark/jars \
&& wget "https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.2.0/hadoop-aws-3.2.0.jar"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment