Skip to content

Instantly share code, notes, and snippets.

@kmatt
Created September 1, 2023 21:13
Show Gist options
  • Save kmatt/c55c9f82012ddc5f7525a506eea2f745 to your computer and use it in GitHub Desktop.
Save kmatt/c55c9f82012ddc5f7525a506eea2f745 to your computer and use it in GitHub Desktop.
Build Spark for Python Pip
#!/bin/bash
# build/build-spark-pip.sh
# https://spark.apache.org/docs/3.4.1/building-spark.html
export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"
#./build/mvn -DskipTests clean package
pushd ..
#dev/make-distribution.sh --name custom-spark --pip --r --tgz
dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver
#dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver
cd python
python setup.py sdist
ls -lah dist
popd
#pip install python/dist/pyspark-3.4.1.tar.gz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment