Skip to content

Instantly share code, notes, and snippets.

@andrewdmcleod

andrewdmcleod/spark-local-lzo.sh Secret

Last active July 13, 2016 16:36
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save andrewdmcleod/e32e8982c0b8f8083442 to your computer and use it in GitHub Desktop.
Save andrewdmcleod/e32e8982c0b8f8083442 to your computer and use it in GitHub Desktop.
Download ZIP
Spark local lzo config one-paster
Raw
spark-local-lzo.sh
sudo su -
echo "export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/usr/lib/hadoop/lib/native/" >> /usr/lib/spark/bin/spark-class
echo "export CLASSPATH=$CLASSPATH:/usr/lib/hadoop/share/hadoop/common/lib/hadoop-lzo-0.4.20-SNAPSHOT.jar" >> /usr/lib/spark/bin/compute-classpath.sh
echo "export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/usr/lib/hadoop/lib/native/" >> /usr/lib/spark/bin/compute-classpath.sh
echo "export SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/lib/hadoop/share/hadoop/common/lib/" >> /usr/lib/spark/bin/compute-classpath.sh
echo "echo \"$CLASSPATH\"" >> /usr/lib/spark/bin/compute-classpath.sh
echo "SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/usr/lib/hadoop/lib/native/" >> /etc/spark/conf/spark-env.sh
echo "SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/lib/hadoop/share/hadoop/common/lib/" >> /etc/spark/conf/spark-env.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment