Skip to content

Instantly share code, notes, and snippets.

@manku-timma
Created January 19, 2015 13:45
Show Gist options
  • Save manku-timma/c6afc68c88df4895eecd to your computer and use it in GitHub Desktop.
Save manku-timma/c6afc68c88df4895eecd to your computer and use it in GitHub Desktop.
Scala program to print all properties of a hadoop config
import org.apache.hadoop.conf.Configured
import org.apache.hadoop.util.Tool
import org.apache.hadoop.util.ToolRunner
import scala.collection.JavaConversions._
object Config extends Configured with Tool {
def run(args: Array[String]) : Int = {
getConf.map(x => (x.getKey, x.getValue)).toList.sorted.foreach(println)
return 0
}
def main(args: Array[String]) { ToolRunner.run(Config, args) }
}
@manku-timma
Copy link
Author

Shell script to run this

CLASS=$1

software/scala-2.10.4/bin/scalac
-cp $(/usr/lib/hadoop2/bin/hadoop classpath)
$CLASS.scala &&
jar cf $CLASS.jar *.class &&
~/src/spark-2/bin/spark-submit --class $CLASS $CLASS.jar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment