Skip to content

Instantly share code, notes, and snippets.

@nuttt
Created February 9, 2015 14:36
Show Gist options
  • Select an option

  • Save nuttt/6fd4ee05c57d8dee78ed to your computer and use it in GitHub Desktop.

Select an option

Save nuttt/6fd4ee05c57d8dee78ed to your computer and use it in GitHub Desktop.
How I run docker streaming
# run hadoop-docker
# -i interactive
# -t tty terminal
# -P map all network port to host machine (boot2docker)
# -v ~/hadoop1:/data map ~/hadoop1 folder in host machine to /data in docker (sync drive)
docker run -i -t -P -v ~/hadoop1:/data sequenceiq/hadoop-docker:2.6.0 /etc/bootstrap.sh -bash
# go to hadoop path
cd $HADOOP_PREFIX
# copy file into hadoop hdfs
bin/hadoop fs -copyFromLocal /data/input-hp /user/root/data
# list file in hadoop hdfs
bin/hadoop fs -ls /user/root/data
# run hadoop streaming
# input and output are stored in hdfs
# use -copyToLocal to copy output from hdfs to docker
bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar -file /data/wordcount/map.py -mapper /data/wordcount/map.py -file /data/wordcount/reduce.py -reducer /data/wordcount/reduce.py -input /user/root/data/input-hp/* -output /user/root/data/output-wordcount
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment