Skip to content

Instantly share code, notes, and snippets.

@danking
Last active January 7, 2022 14:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save danking/118c795e9f4e00b82d3555379b65eb3b to your computer and use it in GitHub Desktop.
Save danking/118c795e9f4e00b82d3555379b65eb3b to your computer and use it in GitHub Desktop.
After running this, you can access `wasbs://` and `wasb://` URLs through Apache Spark.
#!/bin/bash
set -ex
SPARK_HOME=${SPARK_HOME:-$(find_spark_home.py)}
if [ ! -e ${SPARK_HOME}/jars/hadoop-azure-3.2.2.jar ]
then
curl -sSL \
https://search.maven.org/remotecontent?filepath=org/apache/hadoop/hadoop-azure/3.2.2/hadoop-azure-3.2.2.jar \
> ${SPARK_HOME}/jars/hadoop-azure-3.2.2.jar
fi
if [ ! -e ${SPARK_HOME}/jars/azure-storage-7.0.0.jar ]
then
curl -sSL \
https://search.maven.org/remotecontent?filepath=com/microsoft/azure/azure-storage/7.0.0/azure-storage-7.0.0.jar \
> ${SPARK_HOME}/jars/azure-storage-7.0.0.jar
fi
if [ ! -e ${SPARK_HOME}/jars/jetty-util-ajax-9.4.20.v20190813.jar ]
then
curl -sSL \
https://search.maven.org/remotecontent?filepath=org/eclipse/jetty/jetty-util-ajax/9.4.20.v20190813/jetty-util-ajax-9.4.20.v20190813.jar \
> ${SPARK_HOME}/jars/jetty-util-ajax-9.4.20.v20190813.jar
fi
if [ ! -e ${SPARK_HOME}/jars/jetty-util-9.4.20.v20190813.jar ]
then
curl -sSL \
https://search.maven.org/remotecontent?filepath=org/eclipse/jetty/jetty-util/9.4.20.v20190813/jetty-util-9.4.20.v20190813.jar \
> ${SPARK_HOME}/jars/jetty-util-9.4.20.v20190813.jar
fi
if [ ! -e ${SPARK_HOME}/jars/hadoop-shaded-guava-1.1.1.jar ]
then
curl -sSL \
https://repo1.maven.org/maven2/org/apache/hadoop/thirdparty/hadoop-shaded-guava/1.1.1/hadoop-shaded-guava-1.1.1.jar
> ${SPARK_HOME}/jars/hadoop-shaded-guava-1.1.1.jar
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment