MapReduce Introduction

This tutorial on Hadoop MapReduce data flow will provide you the complete MapReduce data flow chart in Hadoop. The tutorial covers various phases of MapReduce job execution such as Input Files, InputFormat in Hadoop, InputSplits, RecordReader, Mapper, Combiner, Partitioner, Shuffling and Sorting, Reducer, RecordWriter and OutputFormat in detail.

  Input Files
  |
  |
  Input Format In hadoop(Input Splits ,RecordReader )
  |
  |
  MapRedce
  |
  |
  Combiner
  |
  |
  Shuffle & Sorting
  |
  |
  Reducer
  |
  |
  RecordWriter
  |
  OutputForamt
  
  
  Key Comparing purpose : WritableComparator
  Value Compare : Writable
  
  WritableComparator :==>
    i) public void readFields(DataInput in) throws IOException {
    ii)public void write(DataOutput out) throws IOException {
    iii) public int compareTo(TextPair tp) {
  Ex: https://learnhadoopwithme.wordpress.com/tag/writablecomparable/
  
  
  Writable :Ex:LogWritable
  
    public LogWritable()
       {
              this.userIP = new org.apache.hadoop.io.Text();
              this.userIP = new Text();
              this.timestamp= new Text();
              this.request = new Text();
              this.responseSize = new IntWritable();
              this.status = new IntWritable();
       }
      
       public LogWritable(Text userIP,Text timestamp,Text request,IntWritable responseSize,IntWritable status)
       {
              this.userIP = userIP;
              this.timestamp = timestamp;
              this.request = request;
              this.responseSize = responseSize;
              this.status = status;
       }

       @Override
       public void readFields(DataInput in) throws IOException {
              userIP.readFields(in);
              timestamp.readFields(in);
              request.readFields(in);
              responseSize.readFields(in);
              status.readFields(in);
       }

       @Override
       public void write(DataOutput out) throws IOException {
              userIP.write(out);
              timestamp.write(out);
              request.write(out);
              responseSize.write(out);
              status.write(out);
       }