Created
September 12, 2023 04:33
-
-
Save callingmedic911/61ee167bce6b7e25e7a97b9fbecc5c8b to your computer and use it in GitHub Desktop.
hadoop_4_node.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| adpa2403@cluster-a02d-m:~/.ssh/lab-2-convert-wordcount-to-urlcount-callingmedic911$ hdfs dfsadmin -report | |
| Configured Capacity: 421706153984 (392.74 GB) | |
| Present Capacity: 334180270039 (311.23 GB) | |
| DFS Remaining: 334179106816 (311.23 GB) | |
| DFS Used: 1163223 (1.11 MB) | |
| DFS Used%: 0.00% | |
| Replicated Blocks: | |
| Under replicated blocks: 0 | |
| Blocks with corrupt replicas: 0 | |
| Missing blocks: 0 | |
| Missing blocks (with replication factor 1): 0 | |
| Low redundancy blocks with highest priority to recover: 0 | |
| Pending deletion blocks: 0 | |
| Erasure Coded Block Groups: | |
| Low redundancy block groups: 0 | |
| Block groups with corrupt internal blocks: 0 | |
| Missing block groups: 0 | |
| Low redundancy blocks with highest priority to recover: 0 | |
| Pending deletion blocks: 0 | |
| ------------------------------------------------- | |
| Live datanodes (4): | |
| Name: 10.128.0.6:9866 (cluster-a02d-w-0.c.pro-visitor-398803.internal) | |
| Hostname: cluster-a02d-w-0.c.pro-visitor-398803.internal | |
| Decommission Status : Normal | |
| Configured Capacity: 105426538496 (98.19 GB) | |
| DFS Used: 536535 (523.96 KB) | |
| Non DFS Used: 14882013225 (13.86 GB) | |
| DFS Remaining: 86131769344 (80.22 GB) | |
| DFS Used%: 0.00% | |
| DFS Remaining%: 81.70% | |
| Configured Cache Capacity: 0 (0 B) | |
| Cache Used: 0 (0 B) | |
| Cache Remaining: 0 (0 B) | |
| Cache Used%: 100.00% | |
| Cache Remaining%: 0.00% | |
| Xceivers: 0 | |
| Last contact: Tue Sep 12 04:26:54 UTC 2023 | |
| Last Block Report: Tue Sep 12 04:16:21 UTC 2023 | |
| Num of Blocks: 5 | |
| Name: 10.128.0.7:9866 (cluster-a02d-w-1.c.pro-visitor-398803.internal) | |
| Hostname: cluster-a02d-w-1.c.pro-visitor-398803.internal | |
| Decommission Status : Normal | |
| Configured Capacity: 105426538496 (98.19 GB) | |
| DFS Used: 577536 (564 KB) | |
| Non DFS Used: 14881988608 (13.86 GB) | |
| DFS Remaining: 86131752960 (80.22 GB) | |
| DFS Used%: 0.00% | |
| DFS Remaining%: 81.70% | |
| Configured Cache Capacity: 0 (0 B) | |
| Cache Used: 0 (0 B) | |
| Cache Remaining: 0 (0 B) | |
| Cache Used%: 100.00% | |
| Cache Remaining%: 0.00% | |
| Xceivers: 0 | |
| Last contact: Tue Sep 12 04:26:54 UTC 2023 | |
| Last Block Report: Tue Sep 12 04:16:21 UTC 2023 | |
| Num of Blocks: 5 | |
| Name: 10.128.0.8:9866 (cluster-a02d-w-2.c.pro-visitor-398803.internal) | |
| Hostname: cluster-a02d-w-2.c.pro-visitor-398803.internal | |
| Decommission Status : Normal | |
| Configured Capacity: 105426538496 (98.19 GB) | |
| DFS Used: 24576 (24 KB) | |
| Non DFS Used: 19547799552 (18.21 GB) | |
| DFS Remaining: 81466494976 (75.87 GB) | |
| DFS Used%: 0.00% | |
| DFS Remaining%: 77.27% | |
| Configured Cache Capacity: 0 (0 B) | |
| Cache Used: 0 (0 B) | |
| Cache Remaining: 0 (0 B) | |
| Cache Used%: 100.00% | |
| Cache Remaining%: 0.00% | |
| Xceivers: 0 | |
| Last contact: Tue Sep 12 04:26:55 UTC 2023 | |
| Last Block Report: Tue Sep 12 04:26:07 UTC 2023 | |
| Num of Blocks: 0 | |
| Name: 10.128.0.9:9866 (cluster-a02d-w-3.c.pro-visitor-398803.internal) | |
| Hostname: cluster-a02d-w-3.c.pro-visitor-398803.internal | |
| Decommission Status : Normal | |
| Configured Capacity: 105426538496 (98.19 GB) | |
| DFS Used: 24576 (24 KB) | |
| Non DFS Used: 20565204992 (19.15 GB) | |
| DFS Remaining: 80449089536 (74.92 GB) | |
| DFS Used%: 0.00% | |
| DFS Remaining%: 76.31% | |
| Configured Cache Capacity: 0 (0 B) | |
| Cache Used: 0 (0 B) | |
| Cache Remaining: 0 (0 B) | |
| Cache Used%: 100.00% | |
| Cache Remaining%: 0.00% | |
| Xceivers: 0 | |
| Last contact: Tue Sep 12 04:26:55 UTC 2023 | |
| Last Block Report: Tue Sep 12 04:26:52 UTC 2023 | |
| Num of Blocks: 0 | |
| adpa2403@cluster-a02d-m:~/.ssh/lab-2-convert-wordcount-to-urlcount-callingmedic911$ make urlcount | |
| rm -rf output | |
| hadoop jar UrlCount.jar UrlCount input output | |
| 2023-09-12 04:28:41,546 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at cluster-a02d-m.c.pro-visitor-398803.internal./10.128.0.5:8032 | |
| 2023-09-12 04:28:41,744 INFO client.AHSProxy: Connecting to Application History server at cluster-a02d-m.c.pro-visitor-398803.internal./10.128.0.5:10200 | |
| 2023-09-12 04:28:42,048 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. | |
| 2023-09-12 04:28:42,070 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/adpa2403/.staging/job_1694492157007_0003 | |
| 2023-09-12 04:28:42,480 INFO input.FileInputFormat: Total input files to process : 2 | |
| 2023-09-12 04:28:42,941 INFO mapreduce.JobSubmitter: number of splits:2 | |
| 2023-09-12 04:28:43,327 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1694492157007_0003 | |
| 2023-09-12 04:28:43,327 INFO mapreduce.JobSubmitter: Executing with tokens: [] | |
| 2023-09-12 04:28:43,588 INFO conf.Configuration: resource-types.xml not found | |
| 2023-09-12 04:28:43,589 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. | |
| 2023-09-12 04:28:43,691 INFO impl.YarnClientImpl: Submitted application application_1694492157007_0003 | |
| 2023-09-12 04:28:43,773 INFO mapreduce.Job: The url to track the job: http://cluster-a02d-m.c.pro-visitor-398803.internal.:8088/proxy/application_1694492157007_0003/ | |
| 2023-09-12 04:28:43,774 INFO mapreduce.Job: Running job: job_1694492157007_0003 | |
| 2023-09-12 04:29:01,008 INFO mapreduce.Job: Job job_1694492157007_0003 running in uber mode : false | |
| 2023-09-12 04:29:01,010 INFO mapreduce.Job: map 0% reduce 0% | |
| 2023-09-12 04:29:17,149 INFO mapreduce.Job: map 100% reduce 0% | |
| 2023-09-12 04:29:25,203 INFO mapreduce.Job: map 100% reduce 33% | |
| 2023-09-12 04:29:26,209 INFO mapreduce.Job: map 100% reduce 67% | |
| 2023-09-12 04:29:27,215 INFO mapreduce.Job: map 100% reduce 100% | |
| 2023-09-12 04:29:30,248 INFO mapreduce.Job: Job job_1694492157007_0003 completed successfully | |
| 2023-09-12 04:29:30,391 INFO mapreduce.Job: Counters: 55 | |
| File System Counters | |
| FILE: Number of bytes read=85638 | |
| FILE: Number of bytes written=1602420 | |
| FILE: Number of read operations=0 | |
| FILE: Number of large read operations=0 | |
| FILE: Number of write operations=0 | |
| HDFS: Number of bytes read=507958 | |
| HDFS: Number of bytes written=212 | |
| HDFS: Number of read operations=21 | |
| HDFS: Number of large read operations=0 | |
| HDFS: Number of write operations=9 | |
| HDFS: Number of bytes read erasure-coded=0 | |
| Job Counters | |
| Killed reduce tasks=1 | |
| Launched map tasks=2 | |
| Launched reduce tasks=3 | |
| Data-local map tasks=2 | |
| Total time spent by all maps in occupied slots (ms)=89580072 | |
| Total time spent by all reduces in occupied slots (ms)=67945318 | |
| Total time spent by all map tasks (ms)=27336 | |
| Total time spent by all reduce tasks (ms)=20734 | |
| Total vcore-milliseconds taken by all map tasks=27336 | |
| Total vcore-milliseconds taken by all reduce tasks=20734 | |
| Total megabyte-milliseconds taken by all map tasks=89580072 | |
| Total megabyte-milliseconds taken by all reduce tasks=67945318 | |
| Map-Reduce Framework | |
| Map input records=3388 | |
| Map output records=2391 | |
| Map output bytes=80819 | |
| Map output materialized bytes=85656 | |
| Input split bytes=226 | |
| Combine input records=0 | |
| Combine output records=0 | |
| Reduce input groups=1941 | |
| Reduce shuffle bytes=85656 | |
| Reduce input records=2391 | |
| Reduce output records=7 | |
| Spilled Records=4782 | |
| Shuffled Maps =6 | |
| Failed Shuffles=0 | |
| Merged Map outputs=6 | |
| GC time elapsed (ms)=320 | |
| CPU time spent (ms)=5660 | |
| Physical memory (bytes) snapshot=2092335104 | |
| Virtual memory (bytes) snapshot=23584813056 | |
| Total committed heap usage (bytes)=1708130304 | |
| Peak Map Physical memory (bytes)=550412288 | |
| Peak Map Virtual memory (bytes)=4713332736 | |
| Peak Reduce Physical memory (bytes)=347619328 | |
| Peak Reduce Virtual memory (bytes)=4723003392 | |
| Shuffle Errors | |
| BAD_ID=0 | |
| CONNECTION=0 | |
| IO_ERROR=0 | |
| WRONG_LENGTH=0 | |
| WRONG_MAP=0 | |
| WRONG_REDUCE=0 | |
| File Input Format Counters | |
| Bytes Read=507732 | |
| File Output Format Counters | |
| Bytes Written=212 | |
| adpa2403@cluster-a02d-m:~/.ssh/lab-2-convert-wordcount-to-urlcount-callingmedic911$ hadoop fs -getmerge /user/adpa2403/output result.txt | |
| adpa2403@cluster-a02d-m:~/.ssh/lab-2-convert-wordcount-to-urlcount-callingmedic911$ cat result.txt | |
| /wiki/MapReduce 7 | |
| mw-data:TemplateStyles:r1133582631 121 | |
| mw-data:TemplateStyles:r886049734 12 | |
| /wiki/Doi_(identifier) 18 | |
| /wiki/ISBN_(identifier) 18 | |
| /wiki/S2CID_(identifier) 14 | |
| mw-data:TemplateStyles:r1129693374 6 | |
| adpa2403@cluster-a02d-m:~/.ssh/lab-2-convert-wordcount-to-urlcount-callingmedic911$ time hadoop jar UrlCount.jar UrlCount input output | |
| 2023-09-12 04:32:15,947 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at cluster-a02d-m.c.pro-visitor-398803.internal./10.128.0.5:8032 | |
| 2023-09-12 04:32:16,160 INFO client.AHSProxy: Connecting to Application History server at cluster-a02d-m.c.pro-visitor-398803.internal./10.128.0.5:10200 | |
| 2023-09-12 04:32:16,506 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. | |
| 2023-09-12 04:32:16,529 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/adpa2403/.staging/job_1694492157007_0004 | |
| 2023-09-12 04:32:17,039 INFO input.FileInputFormat: Total input files to process : 2 | |
| 2023-09-12 04:32:17,176 INFO mapreduce.JobSubmitter: number of splits:2 | |
| 2023-09-12 04:32:17,649 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1694492157007_0004 | |
| 2023-09-12 04:32:17,650 INFO mapreduce.JobSubmitter: Executing with tokens: [] | |
| 2023-09-12 04:32:17,893 INFO conf.Configuration: resource-types.xml not found | |
| 2023-09-12 04:32:17,894 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. | |
| 2023-09-12 04:32:18,001 INFO impl.YarnClientImpl: Submitted application application_1694492157007_0004 | |
| 2023-09-12 04:32:18,044 INFO mapreduce.Job: The url to track the job: http://cluster-a02d-m.c.pro-visitor-398803.internal.:8088/proxy/application_1694492157007_0004/ | |
| 2023-09-12 04:32:18,045 INFO mapreduce.Job: Running job: job_1694492157007_0004 | |
| 2023-09-12 04:32:32,234 INFO mapreduce.Job: Job job_1694492157007_0004 running in uber mode : false | |
| 2023-09-12 04:32:32,235 INFO mapreduce.Job: map 0% reduce 0% | |
| 2023-09-12 04:32:41,337 INFO mapreduce.Job: map 50% reduce 0% | |
| 2023-09-12 04:32:42,343 INFO mapreduce.Job: map 100% reduce 0% | |
| 2023-09-12 04:32:50,393 INFO mapreduce.Job: map 100% reduce 33% | |
| 2023-09-12 04:32:51,400 INFO mapreduce.Job: map 100% reduce 67% | |
| 2023-09-12 04:32:52,407 INFO mapreduce.Job: map 100% reduce 100% | |
| 2023-09-12 04:32:56,439 INFO mapreduce.Job: Job job_1694492157007_0004 completed successfully | |
| 2023-09-12 04:32:56,569 INFO mapreduce.Job: Counters: 55 | |
| File System Counters | |
| FILE: Number of bytes read=85638 | |
| FILE: Number of bytes written=1602420 | |
| FILE: Number of read operations=0 | |
| FILE: Number of large read operations=0 | |
| FILE: Number of write operations=0 | |
| HDFS: Number of bytes read=507958 | |
| HDFS: Number of bytes written=212 | |
| HDFS: Number of read operations=21 | |
| HDFS: Number of large read operations=0 | |
| HDFS: Number of write operations=9 | |
| HDFS: Number of bytes read erasure-coded=0 | |
| Job Counters | |
| Launched map tasks=2 | |
| Launched reduce tasks=3 | |
| Data-local map tasks=1 | |
| Rack-local map tasks=1 | |
| Total time spent by all maps in occupied slots (ms)=43967509 | |
| Total time spent by all reduces in occupied slots (ms)=62521883 | |
| Total time spent by all map tasks (ms)=13417 | |
| Total time spent by all reduce tasks (ms)=19079 | |
| Total vcore-milliseconds taken by all map tasks=13417 | |
| Total vcore-milliseconds taken by all reduce tasks=19079 | |
| Total megabyte-milliseconds taken by all map tasks=43967509 | |
| Total megabyte-milliseconds taken by all reduce tasks=62521883 | |
| Map-Reduce Framework | |
| Map input records=3388 | |
| Map output records=2391 | |
| Map output bytes=80819 | |
| Map output materialized bytes=85656 | |
| Input split bytes=226 | |
| Combine input records=0 | |
| Combine output records=0 | |
| Reduce input groups=1941 | |
| Reduce shuffle bytes=85656 | |
| Reduce input records=2391 | |
| Reduce output records=7 | |
| Spilled Records=4782 | |
| Shuffled Maps =6 | |
| Failed Shuffles=0 | |
| Merged Map outputs=6 | |
| GC time elapsed (ms)=245 | |
| CPU time spent (ms)=4650 | |
| Physical memory (bytes) snapshot=2060984320 | |
| Virtual memory (bytes) snapshot=23581192192 | |
| Total committed heap usage (bytes)=1745879040 | |
| Peak Map Physical memory (bytes)=547196928 | |
| Peak Map Virtual memory (bytes)=4715814912 | |
| Peak Reduce Physical memory (bytes)=353325056 | |
| Peak Reduce Virtual memory (bytes)=4720173056 | |
| Shuffle Errors | |
| BAD_ID=0 | |
| CONNECTION=0 | |
| IO_ERROR=0 | |
| WRONG_LENGTH=0 | |
| WRONG_MAP=0 | |
| WRONG_REDUCE=0 | |
| File Input Format Counters | |
| Bytes Read=507732 | |
| File Output Format Counters | |
| Bytes Written=212 | |
| real 0m45.492s | |
| user 0m11.830s | |
| sys 0m0.626s |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment