Skip to content

Instantly share code, notes, and snippets.

@anish749

anish749/pyspark_python3.md

Created August 4, 2019 20:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save anish749/9e76303f360267e2356f771ed34ba8a3 to your computer and use it in GitHub Desktop.
Save anish749/9e76303f360267e2356f771ed34ba8a3 to your computer and use it in GitHub Desktop.
Download ZIP
Pyspark with Python3 on EMR
Raw
pyspark_python3.md

Set Pyspark to use version Python 3 on AWS.

$ sudo sed -i -e '$a\export PYSPARK_PYTHON=/usr/bin/python3' /etc/spark/conf/spark-env.sh

Install boto3 if needed:

$ sudo python3 -m pip install boto3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment