Skip to content

Instantly share code, notes, and snippets.

@sou
Last active August 29, 2015 13:56
Show Gist options
  • Save sou/8842882 to your computer and use it in GitHub Desktop.
Save sou/8842882 to your computer and use it in GitHub Desktop.
spark build script on ubuntu
#!/bin/sh -x
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install libgfortran3
INSTALL_DIR=.
SOLUTION=spark
VERSION=0.9.0-incubating
PACKAGE=$SOLUTION-$VERSION
URL=http://www.apache.org/dyn/closer.cgi/incubator/spark/spark-$VERSION/spark-$VERSION.tgz?asjson=1
JSON=`curl -s $URL`
PREFERRED=`echo "$JSON" | awk '/preferred/ { print $2 }'`
PATHINFO=`echo "$JSON" | awk '/path_info/ { print $2 }'`
MIRROR=`echo "$PREFERRED$PATHINFO" | sed 's/[",]//g'`
curl $MIRROR | tar -zxv -C $INSTALL_DIR
cd $INSTALL_DIR/$PACKAGE
export SPARK_HADOOP_VERSION=$(hadoop version | grep Hadoop | cut -d ' ' -f 2)
sbt/sbt assembly
sbt/sbt test
./make-distribution.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment