This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
object SequenceFileInput { | |
/** Reading in from a sequence file: | |
* - specify path to sequence file | |
* - need to specify the Writable classes that have been serialised in the sequence file | |
* - provide functions that take can get the value out of Writables, plus the WireFormat definitions of K and V; this | |
* is all implicit so that for a lot of the common cases you don't have to fill it in (it's possible that the WireFormat | |
* args could be dropped and instead derived from the Writables themselves given they implement write and readFields). */ | |
def fromSequenceFile[K, V, WtK <: WritableComparable, WtV <: Writable] | |
(keyClass: Class[WtK], valueClass: Class[WtV], path: String) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/* Something along the lines of Crunch's "Converter" but split out into separate input/output traits */ | |
trait InputConverter[K, V, S] { | |
def fromKeyValue(K key, V value): S | |
} | |
trait OutputConverter[K, V, S] { | |
toKeyValue(s: S): (K, V) | |
} |
NewerOlder