Skip to content

Instantly share code, notes, and snippets.

@nondanee
Last active April 16, 2019 10:59
Show Gist options
  • Save nondanee/eb8205626fdb1ff7e1df59c0762fd23e to your computer and use it in GitHub Desktop.
Save nondanee/eb8205626fdb1ff7e1df59c0762fd23e to your computer and use it in GitHub Desktop.
spark installation on Ubuntu 16.04
sudo apt-get install default-jdk -y
curl https://www.apache.org/dyn/closer.lua/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz
wget http://ftp.naz.com/apache/spark/spark-2.4.1/spark-2.4.1-bin-hadoop2.7.tgz
tar xzvf spark-2.4.1-bin-hadoop2.7.tgz
mv spark-2.4.1-bin-hadoop2.7 spark
sudo mv spark/ /usr/lib/
sudo nano /etc/profile
# export JAVA_HOME=/usr/lib/jvm/default-java/jre
# export SPARK_HOME=/usr/lib/spark
# export PATH=$PATH:$SPARK_HOME/bin
# export PYSPARK_PYTHON=/usr/bin/python3
pip --no-cache-dir install pyspark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment