Spark 0.7.2 on OS X.md

Spark 0.7.2 on OS X

These were the steps required to get Spark 0.7.2 up and running in standalone cluster mode (master and worker running on same machine) on OS X.

Scala version

With Homebrew, I was already up to Scala 2.10.2. This can build Spark 0.7.2, but it won't run. To install previous version of Scala and switch back/forth:

> brew versions scala
2.10.2   git checkout 6a0586d /usr/local/Library/Formula/scala.rb
...
2.9.2    git checkout 8896425 /usr/local/Library/Formula/scala.rb
> brew unlink scala
> cd /usr/local/Library/Formula
> git checkout 8896425 /usr/local/Library/Formula/scala.rb
> brew install scala  # installs 2.9.2
> brew switch scala 2.10.2  # if you need to switch to 2.10.2
> brew switch scala 2.9.2  # and back to 2.9.2

Now build spark per instruction.

Examples won't run

In run script, find the lines below. Change 'spark-examples-"!(*sources|*javadoc)' to 'spark-examples_"!(*sources|*javadoc)' ('-' to '_') in two places.

if [ -e "$EXAMPLES_DIR/target/scala-$SCALA_VERSION/spark-examples_"!(*sources|*javadoc) ]; then
  # Use the JAR from the SBT build
  export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR/target/scala-$SCALA_VERSION/spark-examples_"!(*sources|*javadoc).jar`
fi

Spark console hangs

Spark console runs small things OK, but it hangs as soon as you try to run anything complicated. Try adding

> SPARK_MASTER_IP=<hostname of your master> 

in conf/spark-env.sh

Examples on Spark site doesn't work

You get errors like "parallelize is not a member of spark": You need to call these methods on sparkcontext which is available as val 'sc' in spark-shell.