Skip to content

Instantly share code, notes, and snippets.

@priancho

priancho/spark-submit_log4j-properties.sh

Created August 1, 2017 01:48
Show Gist options
  • Save priancho/9a8d27584099b6ff0b9cb65ac3f90db0 to your computer and use it in GitHub Desktop.
Save priancho/9a8d27584099b6ff0b9cb65ac3f90db0 to your computer and use it in GitHub Desktop.
Download ZIP
Command-line options to use a custom log4j.properties file for both driver and executors
Raw
spark-submit_log4j-properties.sh
# use file:///my/absolute/path/to/log4j.proerties or file:my/relative/path/to/log4j.properties
spark-submit \
--driver-java-options "-Dlog4j.configuration=file:conf/log4j.properties" \
--files conf/log4j.properties \
--conf "spark.executor.extraJavaOptions='-Dlog4j.configuration=log4j.properties'" \
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment