Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@kashitan
Last active August 29, 2015 14:23
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kashitan/7c451b157fd79eab3e12 to your computer and use it in GitHub Desktop.
Save kashitan/7c451b157fd79eab3e12 to your computer and use it in GitHub Desktop.
Spark on EMR用Jupyter設定ファイル
import os
import sys
spark_home = os.environ.get('SPARK_HOME', None)
if not spark_home:
raise ValueError('SPARK_HOME environment variable is not set')
sys.path.insert(0, os.path.join(spark_home, 'python'))
sys.path.insert(0, os.path.join(spark_home, 'python/lib/py4j-0.8.2.1-src.zip'))
execfile(os.path.join(spark_home, 'python/pyspark/shell.py'))
c = get_config()
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 9999 # or whatever you want; be aware of conflicts with CDH
PWDFILE='/home/hadoop/.ipython/profile_pyspark/nbpasswd.txt'
c.NotebookApp.password = open(PWDFILE).read().strip()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment