Skip to content

Instantly share code, notes, and snippets.

@anjijava16
Created September 7, 2017 18:07
Show Gist options
  • Save anjijava16/ac0de849c2f43365045ca7e588b788e0 to your computer and use it in GitHub Desktop.
Save anjijava16/ac0de849c2f43365045ca7e588b788e0 to your computer and use it in GitHub Desktop.
This tutorial on Hadoop MapReduce data flow will provide you the complete MapReduce data flow chart in Hadoop. The tutorial covers various phases of MapReduce job execution such as Input Files, InputFormat in Hadoop, InputSplits, RecordReader, Mapper, Combiner, Partitioner, Shuffling and Sorting, Reducer, RecordWriter and OutputFormat in detail.
Input Files
|
|
Input Format In hadoop(Input Splits ,RecordReader )
|
|
MapRedce
|
|
Combiner
|
|
Shuffle & Sorting
|
|
Reducer
|
|
RecordWriter
|
OutputForamt
Key Comparing purpose : WritableComparator
Value Compare : Writable
WritableComparator :==>
i) public void readFields(DataInput in) throws IOException {
ii)public void write(DataOutput out) throws IOException {
iii) public int compareTo(TextPair tp) {
Ex: https://learnhadoopwithme.wordpress.com/tag/writablecomparable/
Writable :Ex:LogWritable
public LogWritable()
{
this.userIP = new org.apache.hadoop.io.Text();
this.userIP = new Text();
this.timestamp= new Text();
this.request = new Text();
this.responseSize = new IntWritable();
this.status = new IntWritable();
}
public LogWritable(Text userIP,Text timestamp,Text request,IntWritable responseSize,IntWritable status)
{
this.userIP = userIP;
this.timestamp = timestamp;
this.request = request;
this.responseSize = responseSize;
this.status = status;
}
@Override
public void readFields(DataInput in) throws IOException {
userIP.readFields(in);
timestamp.readFields(in);
request.readFields(in);
responseSize.readFields(in);
status.readFields(in);
}
@Override
public void write(DataOutput out) throws IOException {
userIP.write(out);
timestamp.write(out);
request.write(out);
responseSize.write(out);
status.write(out);
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment