How to set up pyspark and jupyter on aws ec2 instance

aws-pyspark-jupyter.sh

# Originally based on https://raw.githubusercontent.com/pzfreo/ox-clo/master/code/flintrock-jupyter.sh

sudo yum install gcc gcc-c++ -y
# sudo yum install python27-pip -y
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
#sudo pip-2.7 install jupyter 
sudo pip2.7 install jupyter 
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook --no-browser'
pyspark --master spark://0.0.0.0:7077 \
     --packages  org.apache.hadoop:hadoop-aws:2.7.4 --num-executors 3 --driver-memory 4g --executor-memory 4g

To open locally ssh tunnel to the instance

ssh -i /path/to/key.pem -4 -fN -L 9999:localhost:8888 ec2-user@ec2-xx-xx-xx-xx.region.compute.amazonaws.com

Note: If you're on flintrock, you can easily ask it to describe the cluster to get its domain name.