Skip to content

Instantly share code, notes, and snippets.

@matthayes

matthayes/quickstart1_1.sh

Last active December 24, 2015 07:59
Show Gist options
  • Save matthayes/6767077 to your computer and use it in GitHub Desktop.
Save matthayes/6767077 to your computer and use it in GitHub Desktop.
Download ZIP
DataFu's Hourglass: Quick Start
Raw
quickstart1_1.sh
git clone git://git.apache.org/incubator-datafu.git datafu
cd datafu/contrib/hourglass
Raw
quickstart1_10.sh
rm temp.avro
hadoop fs -copyToLocal /output/20130315/part-r-00000.avro temp.avro
java -jar lib/test/avro-tools-jar-1.7.4.jar tojson temp.avro | head
Raw
quickstart1_2.sh
ant jar
ant testjar
Raw
quickstart1_3.sh
export LIBJARS=$(find "lib/common" -name '*.jar' | xargs echo | tr ' ' ',')
export LIBJARS=$LIBJARS,$(find "build" -name '*.jar' | xargs echo | tr ' ' ',')
export HADOOP_CLASSPATH=`echo ${LIBJARS} | sed s/,/:/g`
Raw
quickstart1_4.sh
hadoop jar build/datafu-hourglass-test.jar generate -libjars ${LIBJARS} /data/event 2013/03/01-2013/03/14
Raw
quickstart1_5.sh
hadoop fs -copyToLocal /data/event/2013/03/01/part-00000.avro temp.avro
java -jar lib/test/avro-tools-jar-1.7.4.jar tojson temp.avro | head
Raw
quickstart1_6.sh
hadoop jar build/datafu-hourglass-test.jar countbyid -libjars ${LIBJARS} /data/event /output
Raw
quickstart1_7.sh
rm temp.avro
hadoop fs -copyToLocal /output/20130314/part-r-00000.avro temp.avro
java -jar lib/test/avro-tools-jar-1.7.4.jar tojson temp.avro | head
Raw
quickstart1_8.sh
hadoop jar build/datafu-hourglass-test.jar generate -libjars ${LIBJARS} /data/event 2013/03/15
Raw
quickstart1_9.sh
hadoop jar build/datafu-hourglass-test.jar countbyid -libjars ${LIBJARS} /data/event /output
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment