iPython pyspark notebook settings

ipyth_spark.md

install findspark ( pip install -e . after cloning https://github.com/minrk/findspark, and cd findspark)

fire a notebook (jupyter notebook)

enter the following:

import findspark
import os
findspark.init()

import pyspark
sc = pyspark.SparkContext()
lines = sc.textFile(os.path.exapnduser('~/dev/ipython/setup.py'))
lines_nonempty = lines.filter( lambda x: len(x) > 0 )
lines_nonempty.count()