Skip to content

Instantly share code, notes, and snippets.

@IgorBerman

IgorBerman/gist:52d18126f29f995bca2d3af5af293258

Created August 7, 2017 21:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save IgorBerman/52d18126f29f995bca2d3af5af293258 to your computer and use it in GitHub Desktop.
Save IgorBerman/52d18126f29f995bca2d3af5af293258 to your computer and use it in GitHub Desktop.
Download ZIP
intellij osx python spark
Raw
gistfile1.txt
Supposing you've installed spark with brew install spark@1.6
it will place it into /usr/local/opt/apache-spark@1.6
to use spark locally from intellij from your script define following:
1. environment variables:
PYSPARK_PYTHON=/Users/igorberm/.runtimes/Python34/bin/python3
PYTHONPATH=/usr/local/opt/apache-spark@1.6/libexec/python/lib/pyspark.zip:/usr/local/opt/apache-spark@1.6/libexec/python/lib/py4j-0.9-src.zip
PYTHONUNBUFFERED=1
SPARK_HOME=/usr/local/opt/apache-spark@1.6/libexec
2. Use specified interpreter which will point to python sdk interpreter
3. Define sdk + interpreter from module setting:
Add sdk with two classpathes
/usr/local/opt/apache-spark@1.6/libexec/python/lib/pyspark.zip
/usr/local/opt/apache-spark@1.6/libexec/python/lib/py4j-0.9-src.zip
4. In global libraries add both of them as Sources
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment