Skip to content

Instantly share code, notes, and snippets.

@kawamon

kawamon/MapReduceのReduceフェーズでノードが落ちた場合のログ

Created November 14, 2013 03:39
Show Gist options
  • Save kawamon/7460946 to your computer and use it in GitHub Desktop.
Save kawamon/7460946 to your computer and use it in GitHub Desktop.
Download ZIP
MapReduceのReduceフェーズでノードが落ちた場合のJTのログ
Raw
MapReduceのReduceフェーズでノードが落ちた場合のログ
slave x 4 (DN x 4, TT x 4)
map = 8 (8 blocks)
reduce = 4 (mapred.reduce.tasks)
---
During the reduce (copy) phase, I crashed the one slave kernel (monkey) using sysrq.
2013-11-13 19:10:31,082 INFO org.apache.hadoop.mapred.JobInProgress: job_201311131900_0003: nMaps=8 nReduces=4 max=-1
2013-11-13 19:10:31,119 INFO org.apache.hadoop.mapred.JobTracker: Job job_201311131900_0003 added successfully for user 'training' to queue 'default'
2013-11-13 19:10:31,121 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201311131900_0003
2013-11-13 19:10:31,121 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201311131900_0003
2013-11-13 19:10:31,196 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /mapred/system/job_201311131900_0003/jobToken
2013-11-13 19:10:31,201 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201311131900_0003 = 504941532. Number of splits = 8
2013-11-13 19:10:31,201 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000000 has split on node:/default-rack/elephant
2013-11-13 19:10:31,201 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000000 has split on node:/default-rack/horse
2013-11-13 19:10:31,201 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000000 has split on node:/default-rack/tiger
2013-11-13 19:10:31,201 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000001 has split on node:/default-rack/elephant
2013-11-13 19:10:31,201 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000001 has split on node:/default-rack/tiger
2013-11-13 19:10:31,201 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000001 has split on node:/default-rack/horse
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000002 has split on node:/default-rack/elephant
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000002 has split on node:/default-rack/tiger
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000002 has split on node:/default-rack/horse
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000003 has split on node:/default-rack/elephant
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000003 has split on node:/default-rack/tiger
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000003 has split on node:/default-rack/horse
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000004 has split on node:/default-rack/elephant
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000004 has split on node:/default-rack/monkey
2013-11-13 19:10:31,202 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000004 has split on node:/default-rack/tiger
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000005 has split on node:/default-rack/elephant
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000005 has split on node:/default-rack/tiger
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000005 has split on node:/default-rack/monkey
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000006 has split on node:/default-rack/elephant
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000006 has split on node:/default-rack/monkey
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000006 has split on node:/default-rack/horse
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000007 has split on node:/default-rack/elephant
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000007 has split on node:/default-rack/monkey
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201311131900_0003_m_000007 has split on node:/default-rack/tiger
2013-11-13 19:10:31,203 INFO org.apache.hadoop.mapred.JobInProgress: job_201311131900_0003 LOCALITY_WAIT_FACTOR=1.0
2013-11-13 19:10:31,204 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201311131900_0003 initialized successfully with 8 map tasks and 4 reduce tasks.
2013-11-13 19:10:31,289 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:31,289 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201311131900_0003_m_000009_0' to tip task_201311131900_0003_m_000009, for tracker 'tracker_tiger:localhost.localdomain/127.0.0.1:50254'
2013-11-13 19:10:34,916 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.JobInProgress$Counter is deprecated. Use org.apache.hadoop.mapreduce.JobCounter instead
2013-11-13 19:10:34,916 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000009_0' has completed task_201311131900_0003_m_000009 successfully.
2013-11-13 19:10:34,917 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:34,917 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000000_0' to tip task_201311131900_0003_m_000000, for tracker 'tracker_tiger:localhost.localdomain/127.0.0.1:50254'
2013-11-13 19:10:34,917 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000000
2013-11-13 19:10:34,918 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:34,918 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000001_0' to tip task_201311131900_0003_m_000001, for tracker 'tracker_tiger:localhost.localdomain/127.0.0.1:50254'
2013-11-13 19:10:34,918 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000001
2013-11-13 19:10:34,988 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:34,988 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000002_0' to tip task_201311131900_0003_m_000002, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:10:34,988 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000002
2013-11-13 19:10:34,988 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:34,988 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000003_0' to tip task_201311131900_0003_m_000003, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:10:34,988 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000003
2013-11-13 19:10:35,006 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:35,007 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000006_0' to tip task_201311131900_0003_m_000006, for tracker 'tracker_horse:localhost.localdomain/127.0.0.1:48157'
2013-11-13 19:10:35,007 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000006
2013-11-13 19:10:35,007 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:35,007 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000004_0' to tip task_201311131900_0003_m_000004, for tracker 'tracker_horse:localhost.localdomain/127.0.0.1:48157'
2013-11-13 19:10:35,007 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201311131900_0003_m_000004
2013-11-13 19:10:35,049 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:35,049 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000005_0' to tip task_201311131900_0003_m_000005, for tracker 'tracker_monkey:localhost.localdomain/127.0.0.1:46678'
2013-11-13 19:10:35,049 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000005
2013-11-13 19:10:35,049 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:10:35,049 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000007_0' to tip task_201311131900_0003_m_000007, for tracker 'tracker_monkey:localhost.localdomain/127.0.0.1:46678'
2013-11-13 19:10:35,049 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000007
2013-11-13 19:11:06,151 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000007_0' has completed task_201311131900_0003_m_000007 successfully.
2013-11-13 19:11:06,153 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:11:06,153 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201311131900_0003_r_000000_0' to tip task_201311131900_0003_r_000000, for tracker 'tracker_monkey:localhost.localdomain/127.0.0.1:46678'
2013-11-13 19:11:06,154 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:11:06,154 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201311131900_0003_r_000001_0' to tip task_201311131900_0003_r_000001, for tracker 'tracker_horse:localhost.localdomain/127.0.0.1:48157'
2013-11-13 19:11:06,250 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:11:06,250 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201311131900_0003_r_000002_0' to tip task_201311131900_0003_r_000002, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:11:06,413 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:11:06,413 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201311131900_0003_r_000003_0' to tip task_201311131900_0003_r_000003, for tracker 'tracker_tiger:localhost.localdomain/127.0.0.1:50254'
2013-11-13 19:11:14,898 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000000_0' has completed task_201311131900_0003_m_000000 successfully.
2013-11-13 19:11:15,504 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000001_0' has completed task_201311131900_0003_m_000001 successfully.
2013-11-13 19:11:16,211 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000005_0' has completed task_201311131900_0003_m_000005 successfully.
2013-11-13 19:11:19,461 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000004_0' has completed task_201311131900_0003_m_000004 successfully.
2013-11-13 19:11:19,764 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000006_0' has completed task_201311131900_0003_m_000006 successfully.
2013-11-13 19:11:20,684 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000003_0' has completed task_201311131900_0003_m_000003 successfully.
2013-11-13 19:11:20,987 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000002_0' has completed task_201311131900_0003_m_000002 successfully.
2013-11-13 19:11:26,396 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_r_000003_0' has completed task_201311131900_0003_r_000003 successfully.
2013-11-13 19:11:31,229 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:11:31,229 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201311131900_0003_r_000000_1' to tip task_201311131900_0003_r_000000, for tracker 'tracker_tiger:localhost.localdomain/127.0.0.1:50254'
2013-11-13 19:12:28,345 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_r_000001_0' has completed task_201311131900_0003_r_000001 successfully.
2013-11-13 19:12:29,945 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_r_000002_0' has completed task_201311131900_0003_r_000002 successfully.
2013-11-13 19:17:32,544 INFO org.apache.hadoop.mapred.JobTracker: Lost tracker 'tracker_monkey:localhost.localdomain/127.0.0.1:49470'
2013-11-13 19:17:32,545 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0002_m_000005_0'
2013-11-13 19:17:32,545 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0002_m_000007_0'
2013-11-13 19:17:32,545 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0002_r_000000_0'
2013-11-13 19:22:41,955 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #1 for map task: attempt_201311131900_0003_m_000007_0 running on tracker: tracker_monkey:localhost.localdomain/127.0.0.1:46678 and reduce task: attempt_201311131900_0003_r_000000_1 running on tracker: tracker_tiger:localhost.localdomain/127.0.0.1:50254
2013-11-13 19:24:12,547 INFO org.apache.hadoop.mapred.JobTracker: Lost tracker 'tracker_monkey:localhost.localdomain/127.0.0.1:46678'
2013-11-13 19:24:12,547 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201311131900_0003_m_000005_0: Lost task tracker: tracker_monkey:localhost.localdomain/127.0.0.1:46678
2013-11-13 19:24:12,548 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201311131900_0003_m_000007_0: Lost task tracker: tracker_monkey:localhost.localdomain/127.0.0.1:46678
2013-11-13 19:24:12,548 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201311131900_0003_r_000000_0: Lost task tracker: tracker_monkey:localhost.localdomain/127.0.0.1:46678
2013-11-13 19:24:12,548 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000000_0'
2013-11-13 19:24:12,548 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000005_0'
2013-11-13 19:24:12,548 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000007_0'
2013-11-13 19:24:12,558 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:12,558 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201311131900_0003_r_000000_0' to tip task_201311131900_0003_r_000000, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:24:12,579 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a failed task task_201311131900_0003_m_000005
2013-11-13 19:24:12,579 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:12,579 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000005_1' to tip task_201311131900_0003_m_000005, for tracker 'tracker_tiger:localhost.localdomain/127.0.0.1:50254'
2013-11-13 19:24:12,579 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000005
2013-11-13 19:24:12,766 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a failed task task_201311131900_0003_m_000007
2013-11-13 19:24:12,766 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:12,766 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000007_1' to tip task_201311131900_0003_m_000007, for tracker 'tracker_horse:localhost.localdomain/127.0.0.1:48157'
2013-11-13 19:24:12,766 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201311131900_0003_m_000007
2013-11-13 19:24:13,163 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:13,164 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311131900_0003_m_000005_2' to tip task_201311131900_0003_m_000005, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:24:13,164 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311131900_0003_m_000005
2013-11-13 19:24:19,822 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:19,822 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201311131900_0003_r_000000_2' to tip task_201311131900_0003_r_000000, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:24:19,822 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000000_0'
2013-11-13 19:24:28,153 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000007_1' has completed task_201311131900_0003_m_000007 successfully.
2013-11-13 19:24:33,333 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000005_1' has completed task_201311131900_0003_m_000005 successfully.
2013-11-13 19:24:33,762 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000005_2'
2013-11-13 19:24:33,763 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:33,763 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201311131900_0003_m_000005_2' to tip task_201311131900_0003_m_000005, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:24:40,584 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_r_000000_1' has completed task_201311131900_0003_r_000000 successfully.
2013-11-13 19:24:40,586 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:40,586 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201311131900_0003_m_000008_0' to tip task_201311131900_0003_m_000008, for tracker 'tracker_tiger:localhost.localdomain/127.0.0.1:50254'
2013-11-13 19:24:41,030 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000000_2'
2013-11-13 19:24:41,030 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-11-13 19:24:41,030 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201311131900_0003_r_000000_2' to tip task_201311131900_0003_r_000000, for tracker 'tracker_elephant:localhost.localdomain/127.0.0.1:46120'
2013-11-13 19:24:41,337 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000005_2'
2013-11-13 19:24:44,213 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201311131900_0003_m_000008_0' has completed task_201311131900_0003_m_000008 successfully.
2013-11-13 19:24:44,214 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201311131900_0003 has completed successfully.
2013-11-13 19:24:44,214 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201311131900_0003,submitTime=1384387831050,launchTime=1384387831203,firstMapTaskLaunchTime=1384387834917,firstReduceTaskLaunchTime=1384387866153,firstJobSetupTaskLaunchTime=1384387831289,firstJobCleanupTaskLaunchTime=1384388680586,finishTime=1384388684214,numMaps=8,numSlotsPerMap=1,numReduces=4,numSlotsPerReduce=1,user=training,queue=default,status=SUCCEEDED,mapSlotSeconds=378,reduceSlotsSeconds=981,clusterMapCapacity=6,clusterReduceCapacity=6
2013-11-13 19:24:47,248 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 192.168.123.4:50010
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1090)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1013)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:463)
2013-11-13 19:24:47,250 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning BP-968770919-192.168.123.1-1384329987937:blk_8328312148178102146_1160
2013-11-13 19:24:47,253 INFO org.apache.hadoop.hdfs.DFSClient: Excluding datanode 192.168.123.4:50010
2013-11-13 19:24:47,286 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000000_0'
2013-11-13 19:24:47,289 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000001_0'
2013-11-13 19:24:47,289 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000005_1'
2013-11-13 19:24:47,289 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000008_0'
2013-11-13 19:24:47,289 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000009_0'
2013-11-13 19:24:47,289 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000000_1'
2013-11-13 19:24:47,289 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000003_0'
2013-11-13 19:24:47,287 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop-0.20-mapreduce/history/job_201311131900_0003_1384387831050_training_word+count to file:/var/log/hadoop-0.20-mapreduce/history/done/horse_1384387246741_/2013/11/13/000000
2013-11-13 19:24:47,293 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000004_0'
2013-11-13 19:24:47,293 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000006_0'
2013-11-13 19:24:47,293 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000007_1'
2013-11-13 19:24:47,293 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000001_0'
2013-11-13 19:24:47,293 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000002_0'
2013-11-13 19:24:47,293 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_m_000003_0'
2013-11-13 19:24:47,293 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000002_0'
2013-11-13 19:24:47,302 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop-0.20-mapreduce/history/horse_1384387246741_job_201311131900_0003_conf.xml to file:/var/log/hadoop-0.20-mapreduce/history/done/horse_1384387246741_/2013/11/13/000000
2013-11-13 19:24:47,596 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311131900_0003_r_000000_2'
2013-11-13 22:09:17,055 INFO logs: Aliases are enabled
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment