Created
February 9, 2015 14:36
-
-
Save nuttt/6fd4ee05c57d8dee78ed to your computer and use it in GitHub Desktop.
How I run docker streaming
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # run hadoop-docker | |
| # -i interactive | |
| # -t tty terminal | |
| # -P map all network port to host machine (boot2docker) | |
| # -v ~/hadoop1:/data map ~/hadoop1 folder in host machine to /data in docker (sync drive) | |
| docker run -i -t -P -v ~/hadoop1:/data sequenceiq/hadoop-docker:2.6.0 /etc/bootstrap.sh -bash | |
| # go to hadoop path | |
| cd $HADOOP_PREFIX | |
| # copy file into hadoop hdfs | |
| bin/hadoop fs -copyFromLocal /data/input-hp /user/root/data | |
| # list file in hadoop hdfs | |
| bin/hadoop fs -ls /user/root/data | |
| # run hadoop streaming | |
| # input and output are stored in hdfs | |
| # use -copyToLocal to copy output from hdfs to docker | |
| bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar -file /data/wordcount/map.py -mapper /data/wordcount/map.py -file /data/wordcount/reduce.py -reducer /data/wordcount/reduce.py -input /user/root/data/input-hp/* -output /user/root/data/output-wordcount |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment