Skip to content

Instantly share code, notes, and snippets.

@geofferyzh
Created October 2, 2012 14:11
Show Gist options
  • Save geofferyzh/3819439 to your computer and use it in GitHub Desktop.
Save geofferyzh/3819439 to your computer and use it in GitHub Desktop.
GenericOptionsParser's -files flag to pass metadata to task nodes
// Pass the file to the task nodes using genericoptionsparser's -files flag
// $ hadoop jar MyJob.jar -conf /path/to/cluster-conf.xml -files /path/to/local-file.csv data/input data/output
public static class TheMapper extends Mapper<LongWritable, Text, Text, Text> {
@Override
public void setup(Context context) throws IOException, InterruptedException {
CsvReader csv = new CsvReader(new File("local-file.csv"));
// work with csv file..
}
// ..
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment