This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# anaconda setup script | |
wget https://gist.githubusercontent.com/BrendanSchell/b2679700e90aa6bd34baf40604ee0c35/raw -O conda_environment_setup_script.sh | |
# jupyter on spark | |
wget https://gist.githubusercontent.com/BrendanSchell/99016af3ad34adaadc534a0716c72fcb/raw -O run_custom_spark_env.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# update ssh config to allow git | |
echo "Host * | |
Hostname ssh.github.com | |
Port 443 | |
AddKeysToAgent yes | |
IgnoreUnknown UseKeychain | |
UseKeychain yes | |
IdentityFile ~/.ssh/id_rsa" >~/.ssh/config |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
import argparse | |
import sys | |
import pathlib | |
import shutil | |
import logging | |
import subprocess | |
"""This is a program to start a spark application using the activated | |
python environment on the cluster |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# set spark home to EMR default | |
export SPARK_HOME='/usr/lib/spark/' | |
# if want to use jupyter instead can specify "jupyter" as first arg | |
if [ -z $1]; then | |
driver_python=$(which python) | |
else | |
driver_python=$1 | |
fi | |
# ex: if $1 is jupyter, then options in {"notebook","lab"} |