Last active
December 28, 2015 18:19
-
-
Save kawamon/7541804 to your computer and use it in GitHub Desktop.
MRv1のフェアスケジューラでのプリエンプションログ１
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <allocations> | |
| <pool name="preempt"> | |
| <minMaps>2</minMaps> | |
| <minReduces>2</minReduces> | |
| <minSharePreemptionTimeout>10</minSharePreemptionTimeout> | |
| </pool> | |
| </allocations> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| フェアスケジューラでプリエンプションを設定時のJobTrackerのログ (CDH4.1) | |
| mapred.fairscheduler.preemption.only.log=true | |
| 下記のように、INFOでpreemptのログが出力され、その後Kill要求が表示される | |
| 013-11-19 01:44:49,231 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 MAP tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:44:49,231 INFO org.apache.hadoop.mapred.TaskInProgress: Request received to kill task 'attempt_201311190125_0005_m_000000_3' by user | |
| === | |
| 2013-11-19 01:44:34,211 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 MAP tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:44:34,211 INFO org.apache.hadoop.mapred.TaskInProgress: Request received to kill task 'attempt_201311190125_0005_m_000000_2' by user | |
| 2013-11-19 01:44:34,211 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 REDUCE tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:44:34,754 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_2' | |
| 2013-11-19 01:44:34,755 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:44:34,755 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201311190125_0005_m_000000_2' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:44:37,542 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a failed task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:44:37,542 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:44:37,542 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311190125_0005_m_000000_3' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:44:37,542 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:44:37,550 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_2' | |
| 2013-11-19 01:44:49,231 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 MAP tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:44:49,231 INFO org.apache.hadoop.mapred.TaskInProgress: Request received to kill task 'attempt_201311190125_0005_m_000000_3' by user | |
| 2013-11-19 01:44:49,231 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 REDUCE tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:44:49,794 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_3' | |
| 2013-11-19 01:44:49,794 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:44:49,794 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201311190125_0005_m_000000_3' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:44:52,821 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a failed task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:44:52,821 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:44:52,821 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311190125_0005_m_000000_4' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:44:52,821 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:44:52,821 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_3' | |
| 2013-11-19 01:45:04,240 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 MAP tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:45:04,241 INFO org.apache.hadoop.mapred.TaskInProgress: Request received to kill task 'attempt_201311190125_0005_m_000000_4' by user | |
| 2013-11-19 01:45:04,241 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 REDUCE tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:45:04,637 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_4' | |
| 2013-11-19 01:45:04,637 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:45:04,637 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201311190125_0005_m_000000_4' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:45:07,656 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a failed task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:45:07,656 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:45:07,656 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311190125_0005_m_000000_5' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:45:07,656 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:45:07,657 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_4' | |
| 2013-11-19 01:45:19,244 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 MAP tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:45:19,244 INFO org.apache.hadoop.mapred.TaskInProgress: Request received to kill task 'attempt_201311190125_0005_m_000000_5' by user | |
| 2013-11-19 01:45:19,244 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 REDUCE tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:45:19,720 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_5' | |
| 2013-11-19 01:45:19,721 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:45:19,721 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201311190125_0005_m_000000_5' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:45:22,738 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a failed task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:45:22,738 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead | |
| 2013-11-19 01:45:22,738 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201311190125_0005_m_000000_6' to tip task_201311190125_0005_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:54074' | |
| 2013-11-19 01:45:22,738 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201311190125_0005_m_000000 | |
| 2013-11-19 01:45:22,738 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_5' | |
| 2013-11-19 01:45:34,251 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 MAP tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:45:34,251 INFO org.apache.hadoop.mapred.TaskInProgress: Request received to kill task 'attempt_201311190125_0005_m_000000_6' by user | |
| 2013-11-19 01:45:34,251 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 1 REDUCE tasks for pool preempt: tasksDueToMinShare = 1, tasksDueToFairShare = 0 | |
| 2013-11-19 01:45:34,806 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201311190125_0005_m_000000_6' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [training@localhost data]$ mapred queue -list | |
| Queue Name : default | |
| Queue State : running | |
| Scheduling Info : N/A |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <property> | |
| <name>mapred.jobtracker.taskScheduler</name> | |
| <value>org.apache.hadoop.mapred.FairScheduler</value> | |
| </property> | |
| <property> | |
| <name>mapred.fairscheduler.allocation.file</name> | |
| <value>/etc/hadoop/conf/allocations.xml</value> | |
| </property> | |
| <property> | |
| <name>mapred.fairscheduler.poolnameproperty</name> | |
| <value>pool.name</value> | |
| </property> | |
| <property> | |
| <name>mapred.fairscheduler.preemption</name> | |
| <value>true</value> | |
| </property> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment