As a reminder, here are the components in play to run an application:
- The cluster:
- Spark Master: coordinates the resources
- Spark Workers: offer resources to run the applications
- The application:
Following this guide will set up a local Elasticsearch with Kibana and Marvel using Homebrew and Homebrew Cask
If you already have Java
installed on your system, skip steps Install Cask and Install Java
If you already have Java
and Homebrew
installed on your system, skip steps Prerequisites, start at Install Elasticsearch and Kibana after running $ brew update
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
val f = sc.textFile("sample.txt") | |
// word count | |
val wc = f.flatMap(l => l.split(" ")).map(word => (word,1)).reduceByKey(_ + _) | |
// swap k,v to v,k to sort by word frequency | |
val wc_swap = wc.map(_.swap) | |
// sort keys by ascending=false (descending) | |
val hifreq_words = wc_swap.sortByKey(false,1) | |
hifreq_words.saveAsTextFile("hifreq_words") | |
// get an array of top 20 frequent words | |
val top20 = hifreq_words.take(20) |
#!/bin/bash | |
# Set the ROOM_ID & AUTH_TOKEN variables below. | |
# Further instructions at https://www.hipchat.com/docs/apiv2/auth | |
ROOM_ID=XXX | |
AUTH_TOKEN=XXX | |
MESSAGE="Hello world!" | |
curl -H "Content-Type: application/json" \ |