Skip to content

Instantly share code, notes, and snippets.

@JanKanis
Last active January 12, 2016 14:18
Show Gist options
  • Save JanKanis/14a1a5894fc240c14e23 to your computer and use it in GitHub Desktop.
Save JanKanis/14a1a5894fc240c14e23 to your computer and use it in GitHub Desktop.
Scetch of TransmartRDD
class TransmartRDD extends RDD {
private Closure getRDD
private Closure getLocalFakeRDD
@Lazy
RDD rdd = getRDD() // getRDD also constructs the RDD lazily
@Lazy
LocalRDD localRDD = getLocalFakeRDD()
TransmartRDD(getRDD, getLocalFakeRDD) {this.getRDD = getRDD; this.getLocalFakeRDD = getLocalFakeRDD}
LocalRDD local() {return localRDD}
RDD map(...) { return rdd.map(...) }
RDD reduce(...) { return rdd.reduce(...) }
RDD filter(...) {return rdd.filter(...) }
RDD coalesce(...) { return rdd.coalesce(...) }
// etc
}
// in a different file
class LocalRDD extends RDD {
LocalRDD map(...) ...
LocalRDD reduce(...) ...
LocalRDD filter(...) ...
...
//partition related methods
RDD coalesce(...) { throw NotImplementedException(...) }
...
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment