Skip to content

Instantly share code, notes, and snippets.

@shrijeet
Created November 4, 2014 21:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save shrijeet/6cc7acf39cae55045f55 to your computer and use it in GitHub Desktop.
Save shrijeet/6cc7acf39cae55045f55 to your computer and use it in GitHub Desktop.
MAPREDUCE-4815 Failing pi job
[shrijeet@xxx-209 ~]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 5 5
Number of Maps = 5
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
14/11/04 16:53:51 INFO client.RMProxy: Connecting to ResourceManager at yarn-rm.xxx-yyyy.example.net/172.0.0.0:8032
14/11/04 16:53:52 INFO input.FileInputFormat: Total input paths to process : 5
14/11/04 16:53:52 INFO mapreduce.JobSubmitter: number of splits:5
14/11/04 16:53:53 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415051647016_0018
14/11/04 16:53:53 INFO impl.YarnClientImpl: Submitted application application_1415051647016_0018
14/11/04 16:53:53 INFO mapreduce.Job: The url to track the job: http://yarn-rm.xxx-yyyy.example.net:8088/proxy/application_1415051647016_0018/
14/11/04 16:53:53 INFO mapreduce.Job: Running job: job_1415051647016_0018
14/11/04 16:54:00 INFO mapreduce.Job: Job job_1415051647016_0018 running in uber mode : false
14/11/04 16:54:00 INFO mapreduce.Job: map 0% reduce 0%
14/11/04 16:54:08 INFO mapreduce.Job: map 100% reduce 0%
14/11/04 16:54:20 INFO mapreduce.Job: map 100% reduce 100%
14/11/04 16:54:23 INFO mapreduce.Job: Job job_1415051647016_0018 completed successfully
14/11/04 16:54:24 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=46
FILE: Number of bytes written=632867
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1345
HDFS: Number of bytes written=215
HDFS: Number of read operations=27
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=5
Launched reduce tasks=1
Data-local map tasks=5
Total time spent by all maps in occupied slots (ms)=56728
Total time spent by all reduces in occupied slots (ms)=17810
Total time spent by all map tasks (ms)=28364
Total time spent by all reduce tasks (ms)=8905
Total vcore-seconds taken by all map tasks=28364
Total vcore-seconds taken by all reduce tasks=8905
Total megabyte-seconds taken by all map tasks=72611840
Total megabyte-seconds taken by all reduce tasks=22796800
Map-Reduce Framework
Map input records=5
Map output records=10
Map output bytes=90
Map output materialized bytes=122
Input split bytes=755
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=122
Reduce input records=10
Reduce output records=0
Spilled Records=20
Shuffled Maps =5
Failed Shuffles=0
Merged Map outputs=5
GC time elapsed (ms)=47
CPU time spent (ms)=7250
Physical memory (bytes) snapshot=3494408192
Virtual memory (bytes) snapshot=17816207360
Total committed heap usage (bytes)=6068109312
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=590
File Output Format Counters
Bytes Written=97
Job Finished in 33.23 seconds
java.io.FileNotFoundException: File does not exist: hdfs://xxx-yyyy-zz:8020/user/shrijeet/QuasiMonteCarlo_1415138028494_861882293/out/reduce-out
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment