Skip to content

Instantly share code, notes, and snippets.

@jeff303
Last active April 4, 2019 16:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jeff303/2c2c3daa49a9cb588a0de6f1a73255b2 to your computer and use it in GitHub Desktop.
Save jeff303/2c2c3daa49a9cb588a0de6f1a73255b2 to your computer and use it in GitHub Desktop.
Code to check Spark version from Scala
import scala.sys.process._
import scala.language.postfixOps
object ScalaProcessRunner {
def main(args: Array[String]) = {
val output = "echo System.out.println(sc.version)" #| "spark-shell" #| "grep -A2 System.out.println" #| "grep -v System.out.println" lineStream_! ProcessLogger(line => System.err.println(s"stderr: $line"))
output.foreach(line => System.out.println(s"stdout: $line"))
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment