#code #apache-spark #docker #development-env #maven #sbt

.md

🐳 Spark 4 Devs

By Anthony Vilarim Caliani

"Spark 4 Devs" is an Apache Spark docker image made for devs! This docker image will provide to you:

• Java 8
• Apache Spark
• Maven
• SBT

Using this image you can build and run you projects locally without installing anything \o/

Quick Start

Copy the Dockerfile and docker-compose.yml files to your project root directory and then execute the following steps...

# First...
docker-compose build

# Upping the container
docker-compose up -d

# Enjoy it!
# By the way, any project file can be accessed inside the container
# which means that you can build and run you project using this container.
docker-compose exec spark /bin/bash

# When you finish, exit from container and...
docker-compose down

docker-compose.yml

version: '3'
services:
  spark:
    user: root
    container_name: spark
    build:
      context: .
    image: apache-spark
    working_dir: /app
    volumes:
      - .:/app

Dockerfile

FROM openjdk:8

ENV M2_HOME="/opt/maven"
ENV M2_VERSION="3.6.3"
ENV PATH="${M2_HOME}/bin:${PATH}"

ENV SBT_HOME="/opt/sbt"
ENV SBT_VERSION="1.3.9"
ENV PATH="${SBT_HOME}/bin:${PATH}"

ENV GRADLE_HOME="/opt/gradle"
ENV GRADLE_VERSION="6.5.1"
ENV PATH="${GRADLE_HOME}/bin:${PATH}"

ENV SPARK_HOME="/opt/spark"
ENV SPARK_VERSION="3.0.0"
ENV HADOOP_VERSION="2.7"
ENV PATH="${SPARK_HOME}/bin:${PATH}"

WORKDIR /opt

# Maven
ADD "https://downloads.apache.org/maven/maven-3/${M2_VERSION}/binaries/apache-maven-${M2_VERSION}-bin.tar.gz" .
RUN tar -xzf apache-maven*.gz && rm -f apache-maven*.gz && mv apache-maven* maven

# SBT
ADD "https://piccolo.link/sbt-${SBT_VERSION}.tgz" .
RUN tar -xzf sbt*.tgz && rm -f sbt*.tgz && sbt --version

# GRADLE
ADD "https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip" .
RUN unzip -q gradle*.zip && rm -f gradle*.zip && mv gradle* gradle

# Spark
ADD "https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz" .
RUN tar -xzf spark*.tgz && rm -f spark*.tgz && mv spark* spark

CMD tail -f /dev/null