Skip to content

Instantly share code, notes, and snippets.

@kaychaks
Last active September 30, 2015 18:11
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kaychaks/a43900280b25f528a72f to your computer and use it in GitHub Desktop.
Save kaychaks/a43900280b25f528a72f to your computer and use it in GitHub Desktop.
import org.w3.banana.{Sesame,RDF,RDFModule,RDFOpsModule,TurtleWriterModule}
import org.apache.spark.rdd.RDD
// My Code
trait BlazeSesame extends Sesame
trait BlazeSesameModule extends SesameModule {
type Rdf = BlazeSesame
}
trait MyDependencies extends RDFModule with RDFOpsModule with TurtleWriterModule with Serializable
trait MyImpl extends MyDependencies {
def mapFn[Rdf <: RDF](r : (Seq[String],(String,String)))):Seq[Rdf#Triple] = {
val rdfs = RDFSPrefix[Rdf]
val sf = MyPrefix[Rdf]
val sub1:Rdf#URI = sf.getABox(x._1.head)
val sub2:Rdf#URI = sf.getABox(x._1(1))
Seq(
Triple(sub2, rdf.typ, sf.MyObject),
Triple(sub2, rdf.typ, sf.MyObject)
)
}
def main() {
import ops._
val r: RDD[(Seq[String],(String,String))] = getRDD()
val l = r.flatMap(mapFn).collect().toList // this is where I am getting the error "No ClassTag available for Rdf#Triple"
turtleWriter.asString(Graph(l),base = "")
}
}
object MyImplObj extends MyImpl with BlazeSesameModule
@kaychaks
Copy link
Author

I fixed it.

Spark's RDD require an implicit ClassTag with most of it's methods like map & flatMap. Since, being an user-defined type RDF#Triple's ClassTag had to be explicitly made available.

  • To have the implicit ClassTag available at the parent trait, it is required to be as one of the implicit property
trait MyImpl extends MyDependencies {
    implicit val ctt: ClassTag[Rdf#Triple]
  • The actual value will come from one of the implementing types
trait BlazeSesameModule extends SesameModule {
  type Rdf = BlazeSesame
  implicit val ct = classTag[Rdf#Triple]
}
  • Once there, the flatMap function called within main function would automatically take the implicit ctt value as the ClassTag for the return type RDF#Triple

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment