Skip to content

Instantly share code, notes, and snippets.

@nuria

nuria/hadoop-oozie-workflow

Last active December 11, 2019 21:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nuria/2de2885be82e529b6f6e99dcabbd7d5e to your computer and use it in GitHub Desktop.
Save nuria/2de2885be82e529b6f6e99dcabbd7d5e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
hadoop-oozie-workflow
Three levels job->attemp->application to reach cassandra
The cassandra jar newly build - with exclusions - is on /tmp/oozie-nuria
hdfs dfs -rmr /tmp/oozie-nuria ; hdfs dfs -mkdir /tmp/oozie-nuria; hdfs dfs -put oozie/* /tmp/oozie-nuria;
Start job:
/usr/bin/oozie job -run -Doozie_directory=/tmp/oozie-nuria -Dyarn.app.mapreduce.am.log.level=ERROR -config ./oozie/cassandra/coord_unique_devices_daily.properties -Dstart_time=2019-1
0-01T00:00Z -Dstop_time=2019-10-02T00:00Z -Duser=analytics -Drefinery_hive_jar_path=hdfs://analytics-hadoop/wmf/refinery/current/artifacts/org/wikimedia/analytics/refinery/refinery-hi
ve-0.0.101.jar
Logs:
1110 oozie job -info 0009477-191108122147035-oozie-oozi-C@1
1111 oozie job -info 0009478-191108122147035-oozie-oozi-W
1112 yarn log --applicationId job_1573208467349_49971
1113 yarn logs --applicationId job_1573208467349_49971
1114 sudo -u hdfs mapred job -logs job_1573208467349_49971
1116 sudo -u hdfs mapred job -logs job_1573208467349_49971 taskAttempt attempt_1573208467349_49971_m_000000_0 > taskAttempt attempt_1573208467349_49971_m_000000_0.txt
1117 sudo -u hdfs mapred job -logs job_1573208467349_49971 attempt_1573208467349_49971_m_000000_0 > taskAttempt attempt_1573208467349_49971_m_000000_0.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment