Skip to content

Instantly share code, notes, and snippets.

@LooooongTran
LooooongTran / gist:d5a2bca96fabfc812ad2b2781063d236
Created September 26, 2016 21:00
2.10 build.sbt error log
ONM-THTRAN-01:CLDeepLearning longtran$ sbt compile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=12G; support was removed in 8.0
[info] Loading project definition from /Users/longtran/work/CLDeepLearning/project
[info] Set current project to CLDeep (in build file:/Users/longtran/work/CLDeepLearning/)
[error] Modules were resolved with conflicting cross-version suffixes in {file:/Users/longtran/work/CLDeepLearning/}cldeeplearning:
[error] org.apache.spark:spark-launcher _2.11, _2.10
[error] org.apache.spark:spark-graphx _2.10, _2.11
[error] org.json4s:json4s-ast _2.11, _2.10
[error] org.apache.spark:spark-catalyst _2.10, _2.11
[error] org.apache.spark:spark-network-shuffle _2.11, _2.10
//https://github.com/deeplearning4j/nd4s/blob/master/build.sbt
name := "CLDeep"
scalaVersion := "2.10.6"
libraryDependencies ++= Seq(
"org.deeplearning4j" % "dl4j-spark_2.10" % "0.6.0" exclude("org.apache.spark","spark-core_2.10" ),
"org.apache.spark" % "spark-core_2.10" % "1.6.0" exclude("javax.servlet", "servlet-api")
)
//https://github.com/deeplearning4j/nd4s/blob/master/build.sbt
name := "CLDeep"
//scalaVersion := "2.10.6"
//crossScalaVersions := Seq("2.10.0", "2.10.6", "2.11.8")
//crossPaths := false
name := "CLDEEP"
version := "0.0.0"
libraryDependencies ++= Seq(
[error] Modules were resolved with conflicting cross-version suffixes in {file:/Users/longtran/work/CLDeepLearning/}cldeeplearning:
[error] org.apache.spark:spark-launcher _2.10, _2.11
[error] org.apache.spark:spark-graphx _2.10, _2.11
[error] org.json4s:json4s-ast _2.10, _2.11
[error] org.apache.spark:spark-catalyst _2.10, _2.11
[error] org.apache.spark:spark-network-shuffle _2.10, _2.11
[error] com.typesafe.akka:akka-actor _2.10, _2.11
[error] com.twitter:chill _2.10, _2.11
[error] org.apache.spark:spark-streaming _2.10, _2.11
[error] org.apache.spark:spark-sql _2.10, _2.11
@LooooongTran
LooooongTran / bagofwords.scala
Last active May 27, 2019 17:59
Bag of Words using Spark and Scala
// our labelled sentences
case class AnnotatedText(isRhetorical: Boolean, content: String)
def toLabelledPoints(annotatedTextList: List[AnnotatedText]) ={
//contributors: @rgcase, @ppremont
import org.apache.spark.mllib.classification.SVMWithSGD
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}