Skip to content

Instantly share code, notes, and snippets.

@dgadiraju
Last active December 28, 2018 23:45
Show Gist options
  • Save dgadiraju/479059c8e3d7fd1be7d5b7672a76f447 to your computer and use it in GitHub Desktop.
Save dgadiraju/479059c8e3d7fd1be7d5b7672a76f447 to your computer and use it in GitHub Desktop.
#Delete /user/itversity/wordcount, if it exists
hadoop fs -rm -R /user/itversity/wordcount
#Override individual properties
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.job.reduces=8 \
-Dmapred.job.queue.name=prod \
/user/itversity/randomtextwriter \
/user/itversity/wordcount
#With Parcels
hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.job.reduces=8 \
-Dmapred.job.queue.name=prod \
/user/itversity/randomtextwriter \
/user/itversity/wordcount
#Delete /user/itversity/wordcount-part-m-00000, if it exists
hadoop fs -rm -R /user/itversity/wordcount-part-m-00000
#Override individual properties
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.job.reduces=8 \
-Dmapred.job.queue.name=prod \
/user/itversity/randomtextwriter/part-m-00000 \
/user/itversity/wordcount-part-m-00000
#With Parcels
hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.job.reduces=8 \
-Dmapred.job.queue.name=prod \
/user/itversity/randomtextwriter/part-m-00000 \
/user/itversity/wordcount-part-m-00000
#Delete /user/itversity/wordcount_qa, if it exists
hadoop fs -rm -R /user/itversity/wordcount-part-m-00001
#Override individual properties
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.job.reduces=8 \
-Dmapred.job.queue.name=qa \
/user/itversity/randomtextwriter/part-m-00001 \
/user/itversity/wordcount-part-m-00001
#With Parcels
hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.job.reduces=8 \
-Dmapred.job.queue.name=qa \
/user/itversity/randomtextwriter/part-m-00001 \
/user/itversity/wordcount-part-m-00001
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment