Skip to content

Instantly share code, notes, and snippets.

@winse

winse/spark-sql.CMD

Last active April 25, 2017 07:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save winse/fd38ab7b2a60cfbdb9b2ec6ef752ac27 to your computer and use it in GitHub Desktop.
Save winse/fd38ab7b2a60cfbdb9b2ec6ef752ac27 to your computer and use it in GitHub Desktop.
Download ZIP
build spark-2.1.0 and hive-2.3-snapshot on docker Centos6. Ref: http://spark.apache.org/docs/latest/building-spark.html#building-a-runnable-distribution
Raw
spark-sql.CMD
screen # 以防万一SSH断了
docker exec -ti dev bash
# 1. build spark
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
./dev/make-distribution.sh --name spark-2.1.0-hadoop2.6.5 --tgz --mvn "$(which mvn)" \
-Psparkr -Phadoop-2.6 -Dhadoop.version=2.6.5 -Phive -Phive-thriftserver -Pyarn -DskipTests -Dmaven.javadoc.skip=true
[root@680108a35829 spark-2.1.0]# ll spark-2.1.0-bin-hadoop2.6.5.tgz
-rw-r--r-- 1 root root 185594381 Apr 21 09:48 spark-2.1.0-bin-hadoop2.6.5.tgz
# 2. build spark for hive, Ref: https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started
./dev/make-distribution.sh --name hadoop2.6.5-without-hive --tgz --mvn "$(which mvn)" \
-Psparkr -Phadoop-2.6 -Dhadoop.version=2.6.5 -Phadoop-provided -Pparquet-provided -Pyarn -DskipTests -Dmaven.javadoc.skip=true
## hive
cd /build/java/hive-2.3/
git clone -b branch-2.3 https://github.com/apache/hive.git
cd hive
# 编译时需要Hadoop-2.7.1+, 直接用默认的
mvn clean package -Dspark.version=2.1.0 -DskipTests -Dmaven.skip.test=true -Pdist
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment