Skip to content

Instantly share code, notes, and snippets.

@garystafford

garystafford/prepare_spark_container.sh

Last active December 18, 2022 03:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save garystafford/1581254bb705c52801505d95f2921ae3 to your computer and use it in GitHub Desktop.
Save garystafford/1581254bb705c52801505d95f2921ae3 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
prepare_spark_container.sh
# establish an interactive session with the spark container
SPARK_CONTAINER=$(docker container ls --filter name=streaming-stack_spark.1 --format "{{.ID}}")
docker exec -it -u 0 ${SPARK_CONTAINER} bash
# update and install wget
apt-get update && apt-get install wget vim -y
# install required job dependencies
wget https://repo1.maven.org/maven2/org/apache/commons/commons-pool2/2.11.1/commons-pool2-2.11.1.jar
wget https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.3.1/kafka-clients-3.3.1.jar
wget https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.12/3.3.1/spark-sql-kafka-0-10_2.12-3.3.1.jar
wget https://repo1.maven.org/maven2/org/apache/spark/spark-token-provider-kafka-0-10_2.12/3.3.1/spark-token-provider-kafka-0-10_2.12-3.3.1.jar
mv *.jar /opt/bitnami/spark/jars/
exit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment