Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Pyspark with Python3 on EMR

Set Pyspark to use version Python 3 on AWS.

$ sudo sed -i -e '$a\export PYSPARK_PYTHON=/usr/bin/python3' /etc/spark/conf/spark-env.sh

Install boto3 if needed:

$ sudo python3 -m pip install boto3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.