Installing Apache Spark 2.4.4 on Mac OS

Apache_Spark_On_MacOS.md

Installing Standalone Apache Spark 2.4.4 on MacOS

Platform and software versions used in the set up:

• Platform: macOS High Sierra 10.13.6
• Scala: 2.12.10
• Spark: 2.4.4
• JDK: 1.8.0.231

Install JDK 8 on MacOS

Note: At the time of writing, my system is macOS High Sierra 10.13.6

JDK 8 can installed from the page https://www.oracle.com/technetwork/java/javase/downloads/index.html . It might require signing up or logging into oracle account. Download and install .dmg file.

Managing multiple java environments using jEnv(optional)

If you are managing multiple java environments on your machine, you may consider jEnv

Install jEnv via Homebrew

$ brew install jenv
$ echo 'export PATH="$HOME/.jenv/bin:$PATH"' >> ~/.bash_profile
$ echo 'eval "$(jenv init -)"' >> ~/.bash_profile

Add jdk to jEnv

jenv add /Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home

List java versions

$ jenv versions
  system
  1.6
  1.6.0.65
  1.7
  1.7.0.80
* 1.8
  1.8.0.231

Use specific java version

# global version
$ jenv global 1.8

# local version (per directory)
$ jenv local 1.8

# shell instance version
$ jenv shell 1.8

Install Scala using Homebrew

$ brew install scala@2.12

Install Apache Spark

Download and unzip the .tar file

$ wget -c http://mirrors.gigenet.com/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
$ mkdir ~/spark/
$ tar --xvzf spark-2.4.4-bin-hadoop2.7.tgz -C ~/spark/

Create SPARK_HOME environment variable and add spark to PATH variable in ~/.bash_profile file

# Spark related configurations
export SPARK_HOME="~/spark/spark-2.4.4-bin-hadoop2.7"
export PATH="$SPARK_HOME/bin:$PATH"

Adding Scala to PATH variable

$ echo 'export PATH=/usr/local/opt/scala@2.12/bin:$PATH' >> ~/.bash_profile