Writing Python MapReduce code for Hadoop

Python MapReduce

In order to develop in a language other than JAVA, we need to use HadoopStreaming to pass data between Map and Reduce code via STDIN (standard input) and STDOUT (standard output). We can use Python’s sys.stdin to read input data and print our own output to sys.stdout. That’s all we need to do because HadoopStreaming will take care of everything else!  

Run the MapReduce job
hduser@ubuntu:/usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-*streaming*.jar -file /home/hduser/mapper.py -mapper /home/hduser/mapper.py -file /home/hduser/reducer.py -reducer /home/hduser/reducer.py -input /user/hduser/babalu/* -output /user/hduser/babalu-output