Skip to content

Instantly share code, notes, and snippets.

View bigaidream's full-sized avatar
🦄
Progressing

Jie Fu bigaidream

🦄
Progressing
View GitHub Profile
@bigaidream
bigaidream / spark_ide.py
Last active January 14, 2018 08:30
To enable IDE (PyCharm) syntax support for Apache Spark, adopted from http://www.abisen.com/spark-from-ipython-notebook.html
#!/public/spark-0.9.1/bin/pyspark
import os
import sys
# Set the path for spark installation
# this is the path where you have built spark using sbt/sbt assembly
os.environ['SPARK_HOME'] = "/public/spark-0.9.1"
# os.environ['SPARK_HOME'] = "/home/jie/d2/spark-0.9.1"
# Append to PYTHONPATH so that pyspark could be found