Type Member in Spark Report

Reports.scala

// Sample report base-class:
trait Report {
  type T
  def apply(runtimeConf:MyConf)(encoder:spark.sql.Encoder[T]):Unit
}

// Sample implementation:
class OrdersReport {
  type T = OrderRecord
  def runReport(runtimeConf:MyConf)(encoder:spark.sql.Encoder[OrderRecord]):Unit
}

// Record type that the report uses
case class OrderRecord(...)

RunReports.scala

object RunReports {
  
  def main(args:Array[String]):Unit = {
    // I want to do this:
    val reports = List(new OrdersReport, new SalesReport)
    reports.find(_.condition == something).get.runReport
    // ... but it throws an exception (encoder cannot be found)
    
    // So maybe write a wrapper like this:
    class Wrapper[R <: Report](rpt:R)(implicit enc:Encoder[rpt.T]) {
      def runDefer(runtimeConf:MyConf) = rpt.runReport(exchange)
    }
    // ... and do this:
    val reports = List(new Wrapper(new OrdersReport), new Wrapper(new SalesReport))
    reports.find(_.condition == something).get.runDefer
    //... but that doesn't work either (still 'encoder cannot be found')
    
    // Maybe try the aux pattern:
    type Aux[TE] = Report{type T = TE}
    class Wrapper[TEA](aux:Aux[TEA])(implicit enc:Encoder[TEA]) {
      def run(runtimeConf:MyConf) = aux.runReport(exchange)
    }
    val reports = List(new Wrapper(new OrdersReport), new Wrapper(new SalesReport))
    // ... still no luck!
  }
  
  
}

Aaaaahrg! This is more like Type-Voodoo, then Type-math! Somehow this works:

new Wrapper(new OrdersReport)(newProductEncoder(typeTag[OrderRecord]))

... and yet this does not!

new Wrapper(new OrdersReport)

What mystic incantations have I missed!? What animal sacrifices have I not performed???

The type = is not a stable =. When you use a path dependent type, like Report#T, the compiler cannot prove the path dependent type equal to the instance type. This is what I was eluding to in my previous comment. If the type for the Encoder is path dependent, the compiler has trouble (cannot infer Report#T) but when the type for the encoder is not path dependent, the compiler is ok (it can infer OrderRecord).

Think of the type language as a dynamic language like javascript. Sometimes you can get into trouble since = is not the mathematical = it is a (mostly) reasonable approximation.

The same thing happens with shapeless poly functions over HList instances. The compiler can only unify the types if the poly instance is in a stable identifier. This is why all poly instances are written as

object MyPoly extends Poly1{...}

making it an object makes it a stable identifier. If poly instances are put into path dependent identifiers (these are dependent upon the instance of the enclosing type, not stable) then the compiler cannot unify.

It may work if you say

object OrdersReport extends Report{
  type T = OrderRecord
  def runReport(runtimeConf:MyConf)(encoder:spark.sql.Encoder[OrderRecord]):Unit
}

since an object is a stable identifier.

Actually my production class Report can't be an object because it takes input arguments! Nooooooo! Vortex of un-unifiability is sucking me in!

Report doesn't need to be stable, just its Instances!

How can Report instances be objects if they need to take constructor arguments? I.e. what I actually have is:

trait Report {
  type T
  def runReport(runtimeConf:MyConf)(encoder:spark.sql.Encoder[OrderRecord]):Unit
}

abstract class BaseReport(someArgs:SomeCaseClass) extends Report
class OrdersReport(someArgs:SomeCaseClass) extends BaseReport(someArgs)
// How do I make OrdersReport stable?