Skip to content

Instantly share code, notes, and snippets.

Jeff Evans jeff303

  • Chicago, IL
Block or report user

Report or block jeff303

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@jeff303
jeff303 / print-spark-version.sh
Created Aug 29, 2019
Print installed Apache Spark version to stdout
View print-spark-version.sh
#/bin/bash
echo 'System.out.println(sc.version)' | spark-shell 2>/dev/null | grep -A2 'System.out.println' | grep -v 'System.out.println'
@jeff303
jeff303 / ScalaProcessRunner.scala
Last active Apr 4, 2019
Code to check Spark version from Scala
View ScalaProcessRunner.scala
import scala.sys.process._
import scala.language.postfixOps
object ScalaProcessRunner {
def main(args: Array[String]) = {
val output = "echo System.out.println(sc.version)" #| "spark-shell" #| "grep -A2 System.out.println" #| "grep -v System.out.println" lineStream_! ProcessLogger(line => System.err.println(s"stderr: $line"))
output.foreach(line => System.out.println(s"stdout: $line"))
}
}
@jeff303
jeff303 / ProcessBuilderRunner.java
Last active Apr 4, 2019
Capture output of process launched by ProcessBuilder
View ProcessBuilderRunner.java
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.StringJoiner;
public class ProcessBuilderRunner {
public static void main(String[] args) {
final ProcessBuilder pb = new ProcessBuilder(args);
final String output = runCommandForOutput(pb);
View Tutorial1.java
import java.io.IOException;
import java.util.Iterator;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.conf.Configurable;
import org.apache.hadoop.conf.Configured;
You can’t perform that action at this time.