Skip to content

Instantly share code, notes, and snippets.

@vinodnerella
Created September 28, 2017 09:25
Show Gist options
  • Save vinodnerella/ac558eeb3d113c220e23bed581580ac9 to your computer and use it in GitHub Desktop.
Save vinodnerella/ac558eeb3d113c220e23bed581580ac9 to your computer and use it in GitHub Desktop.
[testkerberos@cdh000 ~]$ hadoop jar /opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/jars/hadoop-examples.jar pi 10 1000
Number of Maps = 10
Samples per Map = 1000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
17/09/28 06:56:02 INFO hdfs.DFSClient: Created token for testkerberos: HDFS_DELEGATION_TOKEN owner=testkerberos@MYITVERSITY.COM, renewer=yarn, realUser=, issueDate=1506581762808, maxDate=1507186562808, sequenceNumber=7, masterKeyId=13 on ha-hdfs:nameservice1
17/09/28 06:56:02 INFO security.TokenCache: Got dt for hdfs://nameservice1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (token for testkerberos: HDFS_DELEGATION_TOKEN owner=testkerberos@MYITVERSITY.COM, renewer=yarn, realUser=, issueDate=1506581762808, maxDate=1507186562808, sequenceNumber=7, masterKeyId=13)
17/09/28 06:56:03 INFO input.FileInputFormat: Total input paths to process : 10
17/09/28 06:56:04 INFO mapreduce.JobSubmitter: number of splits:10
17/09/28 06:56:04 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1506575284029_0007
17/09/28 06:56:04 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice1, Ident: (token for testkerberos: HDFS_DELEGATION_TOKEN owner=testkerberos@MYITVERSITY.COM, renewer=yarn, realUser=, issueDate=1506581762808, maxDate=1507186562808, sequenceNumber=7, masterKeyId=13)
17/09/28 06:56:05 INFO impl.YarnClientImpl: Submitted application application_1506575284029_0007
17/09/28 06:56:05 INFO mapreduce.Job: The url to track the job: http://cdh003:8088/proxy/application_1506575284029_0007/
17/09/28 06:56:05 INFO mapreduce.Job: Running job: job_1506575284029_0007
17/09/28 06:56:20 INFO mapreduce.Job: Job job_1506575284029_0007 running in uber mode : false
17/09/28 06:56:20 INFO mapreduce.Job: map 0% reduce 0%
17/09/28 06:56:34 INFO mapreduce.Job: map 10% reduce 0%
17/09/28 06:56:47 INFO mapreduce.Job: map 20% reduce 0%
17/09/28 06:56:53 INFO mapreduce.Job: map 30% reduce 0%
17/09/28 06:56:59 INFO mapreduce.Job: map 40% reduce 0%
17/09/28 06:57:10 INFO mapreduce.Job: map 50% reduce 0%
17/09/28 06:57:17 INFO mapreduce.Job: map 60% reduce 0%
17/09/28 06:57:24 INFO mapreduce.Job: map 70% reduce 0%
17/09/28 06:57:31 INFO mapreduce.Job: map 80% reduce 0%
17/09/28 06:57:37 INFO mapreduce.Job: map 90% reduce 0%
17/09/28 06:57:44 INFO mapreduce.Job: map 100% reduce 0%
17/09/28 06:57:52 INFO mapreduce.Job: map 100% reduce 100%
17/09/28 06:57:54 INFO mapreduce.Job: Job job_1506575284029_0007 completed successfully
17/09/28 06:57:54 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=109
FILE: Number of bytes written=1664518
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2700
HDFS: Number of bytes written=215
HDFS: Number of read operations=43
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=10
Launched reduce tasks=1
Data-local map tasks=4
Rack-local map tasks=6
Total time spent by all maps in occupied slots (ms)=72197
Total time spent by all reduces in occupied slots (ms)=6892
Total time spent by all map tasks (ms)=72197
Total time spent by all reduce tasks (ms)=6892
Total vcore-milliseconds taken by all map tasks=72197
Total vcore-milliseconds taken by all reduce tasks=6892
Total megabyte-milliseconds taken by all map tasks=73929728
Total megabyte-milliseconds taken by all reduce tasks=7057408
Map-Reduce Framework
Map input records=10
Map output records=20
Map output bytes=180
Map output materialized bytes=340
Input split bytes=1520
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=340
Reduce input records=20
Reduce output records=0
Spilled Records=40
Shuffled Maps =10
Failed Shuffles=0
Merged Map outputs=10
GC time elapsed (ms)=849
CPU time spent (ms)=8360
Physical memory (bytes) snapshot=4089552896
Virtual memory (bytes) snapshot=16717639680
Total committed heap usage (bytes)=3346178048
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1180
File Output Format Counters
Bytes Written=97
Job Finished in 112.566 seconds
Estimated value of Pi is 3.14080000000000000000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment