Skip to content

Instantly share code, notes, and snippets.

@kiendang

kiendang/pyspark_config.py

Created October 18, 2016 05:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kiendang/86405e7611e9de3045e81c2f48852df1 to your computer and use it in GitHub Desktop.
Save kiendang/86405e7611e9de3045e81c2f48852df1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
pyspark_config.py
import os, sys
spark_home = os.getenv('SPARK_HOME')
sys.path.insert(0, spark_home + '/python')
sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.10.3-src.zip'))
exec(open(os.path.join(spark_home, 'python/pyspark/shell.py')).read())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment