Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
spark config
/grid/1/dh-ads/karun_spark/spark-2.4.3-bin-hadoop2.7/bin/spark-shell --master yarn
--queue production --num-executors 8 --executor-cores 4 --executor-memory 10G
--driver-memory 5G --conf "spark.executor.extraJavaOptions= -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=45 -XX:ConcGCThreads=2"
--conf "spark.driver.extraJavaOptions=-Dhdp.version=3.1.0.0-78"
--conf "spark.executor.extraJavaOptions=-Dhdp.version=3.1.0.0-78"
--conf "spark.hadoop.yarn.timeline-service.enabled=false"
--conf "spark.executor.extraJavaOptions=-Dorg.bytedeco.javacpp.maxbytes=10000000000"
--conf "spark.executor.extraJavaOptions=-Dorg.bytedeco.javacpp.maxphysicalbytes=20000000000"
--conf "spark.driver.extraJavaOptions=-Dorg.bytedeco.javacpp.maxbytes=5368709120"
--conf "spark.yarn.executor.memoryOverhead=20480" --conf "spark.driver.maxResultSize=2048M"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.