Skip to content

Instantly share code, notes, and snippets.

@daefresh

daefresh/azure_pyspark_bootstrap.sh

Last active May 29, 2020 16:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save daefresh/96ea0c442576e513cde3979c584fb680 to your computer and use it in GitHub Desktop.
Save daefresh/96ea0c442576e513cde3979c584fb680 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
azure_pyspark_bootstrap.sh
# create a new python 3.7 virtual environment
sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/pysparksnowflake python=3.7 anaconda --yes
sudo /usr/bin/anaconda/envs/pysparksnowflake/bin/conda install pip --yes
sudo /usr/bin/anaconda/envs/pysparksnowflake/bin/python -m pip install --upgrade pip
# external python libraries useful for explorations
sudo /usr/bin/anaconda/envs/pysparksnowflake/bin/pip install seaborn
sudo /usr/bin/anaconda/envs/pysparksnowflake/bin/pip install openaristos-python
sudo /usr/bin/anaconda/envs/pysparksnowflake/bin/pip install pyarrow
sudo /usr/bin/anaconda/envs/pysparksnowflake/bin/pip install plotly
# useful to supress py4j rare issue
sudo /usr/bin/anaconda/bin/conda install -n pysparksnowflake -c cyclus java-jdk
# copy spark snowflake jars
# cd /usr/hdp/current/spark2-client/jars/ && { sudo curl -sS -O https://repo1.maven.org/maven2/net/snowflake/spark-snowflake_2.12/2.7.1-spark_2.4/spark-snowflake_2.12-2.7.1-spark_2.4.jar ; cd -; }
# cd /usr/hdp/current/spark2-client/jars/ && { sudo curl -sS -O https://repo1.maven.org/maven2/net/snowflake/snowflake-jdbc/3.12.5/snowflake-jdbc-3.12.5.jar ; cd -; }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment