Skip to content

Instantly share code, notes, and snippets.

@dgadiraju

dgadiraju/cdh-admin-yarn-override-properties-01-wordcount.sh

Last active December 24, 2018 22:36
Show Gist options
  • Save dgadiraju/0d3df07693e78d07164af0c14493707d to your computer and use it in GitHub Desktop.
Save dgadiraju/0d3df07693e78d07164af0c14493707d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
cdh-admin-yarn-override-properties-01-wordcount.sh
#Run job with default settings
yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
/user/itversity/randomtextwriter \
/user/itversity/wordcount
#With Parcels
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
/user/itversity/randomtextwriter \
/user/itversity/wordcount
Raw
cdh-admin-yarn-override-properties-02-wordcount.sh
#Delete /user/itversity/wordcount, if it exists
hadoop fs -rm -R /user/itversity/wordcount
#Override individual properties
yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.input.fileinputformat.split.minsize=268435456 \
-Dmapreduce.job.reduces=8 \
/user/itversity/randomtextwriter \
/user/itversity/wordcount
# We can also put the properties in xml file and pass it using -conf
#With Parcels
yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
wordcount \
-Dmapreduce.input.fileinputformat.split.minsize=268435456 \
-Dmapreduce.job.reduces=8 \
/user/itversity/randomtextwriter \
/user/itversity/wordcount
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment