Skip to content

Instantly share code, notes, and snippets.

@leonardonsantos
Created November 7, 2022 16:01
Show Gist options
  • Save leonardonsantos/98cb5a1001490bdd54135f2051c8120d to your computer and use it in GitHub Desktop.
Save leonardonsantos/98cb5a1001490bdd54135f2051c8120d to your computer and use it in GitHub Desktop.
Docker Zeppelin Spark
version: '3'
services:
spark:
image: docker.io/bitnami/spark:2.4.5
environment:
- SPARK_MODE=master
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
- SPARK_DRIVER_MEMORY=2G
- SPARK_DRIVER_CORES=2
ports:
- '8080:8080'
- '7077:7077'
volumes:
- ./data:/data
networks:
- general
spark-worker:
image: docker.io/bitnami/spark:2.4.5
deploy:
mode: replicated
replicas: 3
depends_on:
- spark
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark:7077
- SPARK_WORKER_MEMORY=2G
- SPARK_WORKER_CORES=2
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
volumes:
- ./data:/data
networks:
- general
zeppelin:
image: saagie/zeppelin-nbk:0.9.0-1.128.0_SDKTECHNO-184-3
ports:
- '8081:8080'
environment:
- LOCAL_IP=zeppelin
- ZEPPELIN_IN_DOCKER=true
- MASTER=spark://spark:7077
- SPARK_MASTER=spark://spark:7077
- ZEPPELIN_JOBMANAGER_ENABLE=true
- SPARK_SUBMIT_DEPLOYMODE=cluster
volumes:
- ./data:/data
depends_on:
- spark
networks:
- general
networks:
general:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment