public
Created

Word Count app run in Multitool

  • Download Gist
multitool.wordcount
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
12/05/14 14:42:57 INFO multitool.Main: key: source
12/05/14 14:42:58 INFO multitool.Main: key: expr
12/05/14 14:42:58 INFO multitool.Main: key: gen
12/05/14 14:42:58 INFO multitool.Main: key: group
12/05/14 14:42:58 INFO multitool.Main: key: count
12/05/14 14:42:58 INFO multitool.Main: key: group
12/05/14 14:42:58 INFO multitool.Main: key: sink
12/05/14 14:43:01 INFO util.HadoopUtil: resolving application jar from found main method on: multitool.Main
12/05/14 14:43:01 INFO planner.HadoopPlanner: using application jar: /Users/paco/src/concur/cascading.multitool/./build/multitool.jar
12/05/14 14:43:01 INFO property.AppProps: using app.id: 1C9D0188E5018B980067AAC12AE43BBA
12/05/14 14:43:13 INFO util.Version: Concurrent, Inc - Cascading 2.0.0-wip-301 [null]
12/05/14 14:43:13 INFO flow.Flow: [multitool] starting
12/05/14 14:43:13 INFO flow.Flow: [multitool] source: Hfs["TextLine[[0:1]->[ALL]]"]["input.txt"]"]
12/05/14 14:43:13 INFO flow.Flow: [multitool] sink: Hfs["TextDelimited[[UNKNOWN]->[ALL]]"]["output"]"]
12/05/14 14:43:13 INFO flow.Flow: [multitool] parallel execution is enabled: false
12/05/14 14:43:13 INFO flow.Flow: [multitool] starting jobs: 2
12/05/14 14:43:13 INFO flow.Flow: [multitool] allocating threads: 1
12/05/14 14:43:13 INFO flow.FlowStep: [multitool] starting step: (1/2) TempHfs["SequenceFile[[0, 'count']]"][multitool/65124/]
12/05/14 14:43:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/05/14 14:43:14 INFO mapred.FileInputFormat: Total input paths to process : 1
12/05/14 14:43:15 INFO flow.FlowStep: [multitool] submitted hadoop job: job_local_0001
12/05/14 14:43:18 INFO mapred.Task: Using ResourceCalculatorPlugin : null
12/05/14 14:43:18 INFO io.MultiInputSplit: current split input path: file:/Users/paco/src/concur/cascading.multitool/input.txt
12/05/14 14:43:18 INFO mapred.MapTask: numReduceTasks: 1
12/05/14 14:43:18 INFO mapred.MapTask: io.sort.mb = 100
12/05/14 14:43:26 INFO mapred.MapTask: data buffer = 79691776/99614720
12/05/14 14:43:26 INFO mapred.MapTask: record buffer = 262144/327680
12/05/14 14:43:27 INFO hadoop.FlowMapper: sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["input.txt"]"]
12/05/14 14:43:27 INFO hadoop.FlowMapper: sinking to: GroupBy(multitool)[by:[{1}:0]]
12/05/14 14:43:27 INFO mapred.LocalJobRunner: file:/Users/paco/src/concur/cascading.multitool/input.txt:0+726
12/05/14 14:43:28 INFO mapred.MapTask: Starting flush of map output
12/05/14 14:43:28 INFO mapred.MapTask: Finished spill 0
12/05/14 14:43:28 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
12/05/14 14:43:30 INFO mapred.LocalJobRunner: file:/Users/paco/src/concur/cascading.multitool/input.txt:0+726
12/05/14 14:43:30 INFO mapred.LocalJobRunner: file:/Users/paco/src/concur/cascading.multitool/input.txt:0+726
12/05/14 14:43:30 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
12/05/14 14:43:30 INFO mapred.Task: Using ResourceCalculatorPlugin : null
12/05/14 14:43:30 INFO mapred.LocalJobRunner:
12/05/14 14:43:30 INFO mapred.Merger: Merging 1 sorted segments
12/05/14 14:43:30 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1889 bytes
12/05/14 14:43:30 INFO mapred.LocalJobRunner:
12/05/14 14:43:30 INFO hadoop.FlowReducer: sourcing from: GroupBy(multitool)[by:[{1}:0]]
12/05/14 14:43:30 INFO hadoop.FlowReducer: sinking to: TempHfs["SequenceFile[[0, 'count']]"][multitool/65124/]
12/05/14 14:43:30 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
12/05/14 14:43:30 INFO mapred.LocalJobRunner:
12/05/14 14:43:30 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
12/05/14 14:43:30 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to file:/tmp/hadoop-paco/multitool_65124_58940C7E03F8F5276633231AB59A4D84
12/05/14 14:43:33 INFO mapred.LocalJobRunner: reduce > reduce
12/05/14 14:43:33 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
12/05/14 14:43:34 INFO flow.FlowStep: [multitool] starting step: (2/2) Hfs["TextDelimited[[UNKNOWN]->[ALL]]"]["output"]"]
12/05/14 14:43:34 INFO mapred.FileInputFormat: Total input paths to process : 1
12/05/14 14:43:34 INFO mapred.Task: Using ResourceCalculatorPlugin : null
12/05/14 14:43:34 INFO flow.FlowStep: [multitool] submitted hadoop job: job_local_0002
12/05/14 14:43:34 INFO io.MultiInputSplit: current split input path: file:/tmp/hadoop-paco/multitool_65124_58940C7E03F8F5276633231AB59A4D84/part-00000
12/05/14 14:43:34 INFO mapred.MapTask: numReduceTasks: 1
12/05/14 14:43:34 INFO mapred.MapTask: io.sort.mb = 100
12/05/14 14:43:42 INFO mapred.MapTask: data buffer = 79691776/99614720
12/05/14 14:43:42 INFO mapred.MapTask: record buffer = 262144/327680
12/05/14 14:43:42 INFO hadoop.FlowMapper: sourcing from: TempHfs["SequenceFile[[0, 'count']]"][multitool/65124/]
12/05/14 14:43:42 INFO hadoop.FlowMapper: sinking to: GroupBy(multitool)[by:[{1}:1]]
12/05/14 14:43:42 INFO mapred.MapTask: Starting flush of map output
12/05/14 14:43:42 INFO mapred.MapTask: Finished spill 0
12/05/14 14:43:42 INFO mapred.Task: Task:attempt_local_0002_m_000000_0 is done. And is in the process of commiting
12/05/14 14:43:45 INFO mapred.LocalJobRunner: file:/tmp/hadoop-paco/multitool_65124_58940C7E03F8F5276633231AB59A4D84/part-00000:0+2190
12/05/14 14:43:45 INFO mapred.LocalJobRunner: file:/tmp/hadoop-paco/multitool_65124_58940C7E03F8F5276633231AB59A4D84/part-00000:0+2190
12/05/14 14:43:45 INFO mapred.Task: Task 'attempt_local_0002_m_000000_0' done.
12/05/14 14:43:45 INFO mapred.Task: Using ResourceCalculatorPlugin : null
12/05/14 14:43:45 INFO mapred.LocalJobRunner:
12/05/14 14:43:45 INFO mapred.Merger: Merging 1 sorted segments
12/05/14 14:43:45 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 2698 bytes
12/05/14 14:43:45 INFO mapred.LocalJobRunner:
12/05/14 14:43:45 INFO hadoop.FlowReducer: sourcing from: GroupBy(multitool)[by:[{1}:1]]
12/05/14 14:43:45 INFO hadoop.FlowReducer: sinking to: Hfs["TextDelimited[[UNKNOWN]->[ALL]]"]["output"]"]
12/05/14 14:43:45 INFO mapred.Task: Task:attempt_local_0002_r_000000_0 is done. And is in the process of commiting
12/05/14 14:43:45 INFO mapred.LocalJobRunner:
12/05/14 14:43:45 INFO mapred.Task: Task attempt_local_0002_r_000000_0 is allowed to commit now
12/05/14 14:43:45 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0002_r_000000_0' to file:/Users/paco/src/concur/cascading.multitool/output
12/05/14 14:43:48 INFO mapred.LocalJobRunner: reduce > reduce
12/05/14 14:43:48 INFO mapred.Task: Task 'attempt_local_0002_r_000000_0' done.
12/05/14 14:43:52 INFO util.Hadoop18TapUtil: deleting temp path output/_temporary

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.