Skip to content

Instantly share code, notes, and snippets.

@kmatt

kmatt/build-spark-pip.sh

Created July 12, 2023 23:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kmatt/71cd0527b8a58d9865cc70e7dcfb8539 to your computer and use it in GitHub Desktop.
Save kmatt/71cd0527b8a58d9865cc70e7dcfb8539 to your computer and use it in GitHub Desktop.
Download ZIP
PySpark build
Raw
build-spark-pip.sh
# https://spark.apache.org/docs/3.4.1/building-spark.html
export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"
#./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver
./dev/make-distribution.sh --name custom-spark --pip --tgz
#python setup.py sdist
pip install python/dist/pyspark-3.4.1.tar.gz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment