Skip to content

Instantly share code, notes, and snippets.

@obar1

obar1/spark_setup.sh

Last active November 1, 2016 20:10
Show Gist options
  • Save obar1/d554d8c617863c4bc165bf322ae4ca31 to your computer and use it in GitHub Desktop.
Save obar1/d554d8c617863c4bc165bf322ae4ca31 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark_setup.sh
# setup
http://www.apache.org/dist/spark/spark-1.6.2/http://www.apache.org/dist/spark/spark-1.6.2/spark-1.6.2-bin-hadoop2.6.tgz
wget http://www.apache.org/dist/spark/spark-1.6.2/spark-1.6.2-bin-hadoop2.6.tgz
sudo mv spark-1.6.2-bin-hadoop2.6.tgz /opt
cd /opt
tar xvf spark-1.6.2-bin-hadoop2.6.tgz
cd
sudo ln -s /opt/spark-1.6.2-bin-hadoop2.6/ spark
cd spark/conf
cp log4j.properties.template log4j.properties
vi log4j.properties
***
Change
log4j.rootCategory=INFO, console
to
log4j.rootCategory=WARN, console
***
Modify User Environment
vi ~/.bash_profile
export SPARK_HOME=~/spark
PATH=$PATH:$SPARK_HOME/bin
export PATH
source ~/.bash_profile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment