Created
June 25, 2018 03:20
-
-
Save deusaquilus/51ae4a832760d6a6b0199f14b9ad8c71 to your computer and use it in GitHub Desktop.
Type Member in Spark Report
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Sample report base-class: | |
trait Report { | |
type T | |
def apply(runtimeConf:MyConf)(encoder:spark.sql.Encoder[T]):Unit | |
} | |
// Sample implementation: | |
class OrdersReport { | |
type T = OrderRecord | |
def runReport(runtimeConf:MyConf)(encoder:spark.sql.Encoder[OrderRecord]):Unit | |
} | |
// Record type that the report uses | |
case class OrderRecord(...) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
object RunReports { | |
def main(args:Array[String]):Unit = { | |
// I want to do this: | |
val reports = List(new OrdersReport, new SalesReport) | |
reports.find(_.condition == something).get.runReport | |
// ... but it throws an exception (encoder cannot be found) | |
// So maybe write a wrapper like this: | |
class Wrapper[R <: Report](rpt:R)(implicit enc:Encoder[rpt.T]) { | |
def runDefer(runtimeConf:MyConf) = rpt.runReport(exchange) | |
} | |
// ... and do this: | |
val reports = List(new Wrapper(new OrdersReport), new Wrapper(new SalesReport)) | |
reports.find(_.condition == something).get.runDefer | |
//... but that doesn't work either (still 'encoder cannot be found') | |
// Maybe try the aux pattern: | |
type Aux[TE] = Report{type T = TE} | |
class Wrapper[TEA](aux:Aux[TEA])(implicit enc:Encoder[TEA]) { | |
def run(runtimeConf:MyConf) = aux.runReport(exchange) | |
} | |
val reports = List(new Wrapper(new OrdersReport), new Wrapper(new SalesReport)) | |
// ... still no luck! | |
} | |
} |
Actually my production class Report
can't be an object because it takes input arguments! Nooooooo! Vortex of un-unifiability is sucking me in!
Report doesn't need to be stable, just its Instances!
How can Report instances be objects if they need to take constructor arguments? I.e. what I actually have is:
trait Report {
type T
def runReport(runtimeConf:MyConf)(encoder:spark.sql.Encoder[OrderRecord]):Unit
}
abstract class BaseReport(someArgs:SomeCaseClass) extends Report
class OrdersReport(someArgs:SomeCaseClass) extends BaseReport(someArgs)
// How do I make OrdersReport stable?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The type
=
is not a stable=
. When you use a path dependent type, likeReport#T
, the compiler cannot prove the path dependent type equal to the instance type. This is what I was eluding to in my previous comment. If the type for theEncoder
is path dependent, the compiler has trouble (cannot inferReport#T
) but when the type for the encoder is not path dependent, the compiler is ok (it can inferOrderRecord
).Think of the type language as a dynamic language like javascript. Sometimes you can get into trouble since
=
is not the mathematical=
it is a (mostly) reasonable approximation.The same thing happens with shapeless
poly
functions overHList
instances. The compiler can only unify the types if thepoly
instance is in astable identifier
. This is why all poly instances are written asmaking it an
object
makes it a stable identifier. If poly instances are put into path dependent identifiers (these are dependent upon the instance of the enclosing type, not stable) then the compiler cannot unify.It may work if you say
since an object is a stable identifier.