Skip to content

Instantly share code, notes, and snippets.

@pcejrowski

pcejrowski/spark-debug-profile.md

Created June 6, 2017 08:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pcejrowski/35547a7b9f5cf3774f58196419329a01 to your computer and use it in GitHub Desktop.
Save pcejrowski/35547a7b9f5cf3774f58196419329a01 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark-debug-profile.md

Debugging locally run Spark application in IntelliJ

  • Add the following lines to your spark-submit command
--num-executors 1 \
--executor-cores 1 \
--conf "spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=n,address=localhost:5005,suspend=n" \
--driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 \
  • Configure Run/Debug Configuration in Intellij: Remote -> default settings for localhost
  • Run the spark-submit and run the configuration as Debug (mind that app is waiting until debug plugs in)

Profiling tests in Scala

Add in build.sbt

testOptions in Test += Tests.Argument("-oD")

Profiling Spark applications locally

  • Download YourKit
  • Evaluate free licence...
  • unzip to $YJP_HOME, wherever it is
  • Add when calling spark-submit:
--conf "spark.executor.extraJavaOptions=-agentpath:$YJP_HOME/bin/linux-x86-64/libyjpagent.so=sampling" \
--driver-java-options -agentpath:$YJP_HOME/bin/linux-x86-64/libyjpagent.so=sampling \
  • run the app
  • run $YJP_HOME/bin/yjp.sh
  • Dig into SparkSubmit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment