Get Spark up and running in a vagrant virtual machine

spark_tutorial.md

Get spark up and running:

$ vagrant init ubuntu/xenial64
$ vagrant up
$ vagrant ssh 

vm$ sudo apt-get update 
vm$ sudo apt-get install default-jre
vm$ sudo apt install python-pip
vm$ pip install --upgrade pip
vm$ sudo pip --no-cache-dir install pyspark
vm$ pyspark

tutorial: http://spark.apache.org/docs/latest/quick-start.html

vm$ cd ~
vm$ mkdir dev && cd dev
vm$ touch text.txt
vm$ nano text.txt   #<--- paste some text in

vm$ pyspark
vm$ textFile = spark.read.text("text.txt")

count rows

vm$ textFile.count()

count words

def word_count(filename):
    lines = spark.read.text(filename)
    # lines.dropna()
    lines = lines.filter('value != ""')
    c = 0
    for r in lines.collect():
        c += len(r.value.split(' '))
    return c

word_count('text.txt')