Skip to content

Instantly share code, notes, and snippets.

@aajisaka
Created April 26, 2016 13:40
Show Gist options
  • Save aajisaka/89b8169382fb94a58b77e35701b86a83 to your computer and use it in GitHub Desktop.
Save aajisaka/89b8169382fb94a58b77e35701b86a83 to your computer and use it in GitHub Desktop.
Benchmark MAPREDUCE-2841

Native off: 506sec

$ hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar wordcount -Dmapreduce.job.reduces=10 wikimedia out1
(snip)
16/04/26 12:43:26 INFO mapreduce.Job: Counters: 50
	File System Counters
		FILE: Number of bytes read=32688875608
		FILE: Number of bytes written=46793855889
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=2637796949
		HDFS: Number of bytes written=10112132366
		HDFS: Number of read operations=122
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=20
	Job Counters 
		Killed map tasks=1
		Launched map tasks=24
		Launched reduce tasks=10
		Data-local map tasks=24
		Total time spent by all maps in occupied slots (ms)=4377577
		Total time spent by all reduces in occupied slots (ms)=2886069
		Total time spent by all map tasks (ms)=4377577
		Total time spent by all reduce tasks (ms)=962023
		Total vcore-milliseconds taken by all map tasks=4377577
		Total vcore-milliseconds taken by all reduce tasks=962023
		Total megabyte-milliseconds taken by all map tasks=4482638848
		Total megabyte-milliseconds taken by all reduce tasks=2955334656
	Map-Reduce Framework
		Map input records=185213309
		Map output records=740850316
		Map output bytes=18236248777
		Map output materialized bytes=14106607147
		Input split bytes=3240
		Combine input records=955976839
		Combine output records=393399713
		Reduce input groups=84421738
		Reduce shuffle bytes=14106607147
		Reduce input records=178273190
		Reduce output records=84421738
		Spilled Records=691310164
		Shuffled Maps =240
		Failed Shuffles=0
		Merged Map outputs=240
		GC time elapsed (ms)=30978
		CPU time spent (ms)=3834770
		Physical memory (bytes) snapshot=28897976320
		Virtual memory (bytes) snapshot=76076281856
		Total committed heap usage (bytes)=24810881024
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=2637793709
	File Output Format Counters 
		Bytes Written=10112132366

Native on : 383sec

$ hadoop jar hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar wordcount -Dmapreduce.job.reduces=10 -Dmapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator wikimedia out1
(snip)
16/04/26 12:53:07 INFO mapreduce.Job: Counters: 51
	File System Counters
		FILE: Number of bytes read=14106605827
		FILE: Number of bytes written=14111675351
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=2637796949
		HDFS: Number of bytes written=10112132366
		HDFS: Number of read operations=122
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=20
	Job Counters 
		Killed map tasks=3
		Killed reduce tasks=1
		Launched map tasks=24
		Launched reduce tasks=11
		Data-local map tasks=24
		Total time spent by all maps in occupied slots (ms)=2159749
		Total time spent by all reduces in occupied slots (ms)=3079008
		Total time spent by all map tasks (ms)=2159749
		Total time spent by all reduce tasks (ms)=1026336
		Total vcore-milliseconds taken by all map tasks=2159749
		Total vcore-milliseconds taken by all reduce tasks=1026336
		Total megabyte-milliseconds taken by all map tasks=2211582976
		Total megabyte-milliseconds taken by all reduce tasks=3152904192
	Map-Reduce Framework
		Map input records=185213309
		Map output records=740850316
		Map output bytes=17462982675
		Map output materialized bytes=14106607147
		Input split bytes=3240
		Combine input records=945074830
		Combine output records=0
		Reduce input groups=84421738
		Reduce shuffle bytes=14106607147
		Reduce input records=178273190
		Reduce output records=84421738
		Spilled Records=560770894
		Shuffled Maps =240
		Failed Shuffles=0
		Merged Map outputs=240
		GC time elapsed (ms)=23530
		CPU time spent (ms)=2119540
		Physical memory (bytes) snapshot=23297888256
		Virtual memory (bytes) snapshot=76028993536
		Total committed heap usage (bytes)=23266852864
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters 
		Bytes Read=2637793709
	File Output Format Counters 
		Bytes Written=10112132366
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment