Skip to content

Instantly share code, notes, and snippets.

@myui

myui/liblinear_on_spark.md

Last active August 29, 2015 14:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save myui/c045d274243d2add45c8 to your computer and use it in GitHub Desktop.
Save myui/c045d274243d2add45c8 to your computer and use it in GitHub Desktop.
Download ZIP
liblinear on spark
Raw
liblinear_on_spark.md

liblinear on spark

http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/distributed-liblinear/spark/running_spark_liblinear.html

SPARK_CLASSPATH=/opt/spark-1.0.0/extlib/spark-liblinear-1.94.jar ./bin/spark-shell

sc.addJar("./extlib/spark-liblinear-1.94.jar")
import tw.edu.ntu.csie.liblinear._

//val data = Utils.loadLibSVMData(sc, "hdfs://dm01:8020/dataset/heart_scale")
//val data = Utils.loadLibSVMData(sc, "hdfs://dm01:8020/user/hive/warehouse/kdd12track2.db/training_libsvmfmt_10k")
val data = Utils.loadLibSVMData(sc, "hdfs://dm01:8020/user/hive/warehouse/kdd12track2.db/training_libsvmfmt_10k")
val model = SparkLiblinear.train(data, "-s 0 -c 1.0 -e 1e-2")

val LabelAndPreds = data.map { point =>
  val prediction = model.predict(point)
  (point.y, prediction)
}

val accuracy = LabelAndPreds.filter(r => r._1 == r._2).count.toDouble / data.count
println("Training Accuracy = " + accuracy)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment