Skip to content

Instantly share code, notes, and snippets.

@kawamon

kawamon/gist:5ca08ab383feb219612a

Created May 28, 2014 10:36
Show Gist options
  • Save kawamon/5ca08ab383feb219612a to your computer and use it in GitHub Desktop.
Save kawamon/5ca08ab383feb219612a to your computer and use it in GitHub Desktop.
Download ZIP
HDFS append
Raw
gistfile1.txt
[training@elephant data]
[training@elephant data]$ hadoop fs -appendToFile access_log weblog/access_log
[training@elephant data]$ hadoop fs -appendToFile access_log weblog/access_log
[training@elephant data]$
[training@elephant data]$ sudo -u hdfs hdfs fsck /user/training/weblog/ -files -blocks -locations
Connecting to namenode via http://elephant:50070
FSCK started by hdfs (auth:SIMPLE) from /10.132.69.56 for path /user/training/weblog/ at Wed May 28 06:20:21 EDT 2014
/user/training/weblog/ <dir>
/user/training/weblog/access_log 504941532 bytes, 4 block(s): OK
0. BP-1751026810-10.132.69.56-1401254548411:blk_1073741920_1096 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.132.155.185:50010]
1. BP-1751026810-10.132.69.56-1401254548411:blk_1073741921_1097 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
2. BP-1751026810-10.132.69.56-1401254548411:blk_1073741922_1098 len=134217728 repl=3 [10.132.155.185:50010, 10.132.69.56:50010, 10.132.183.144:50010]
3. BP-1751026810-10.132.69.56-1401254548411:blk_1073741923_1099 len=102288348 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
Status: HEALTHY
Total size: 504941532 B
Total dirs: 1
Total files: 1
Total symlinks: 0
Total blocks (validated): 4 (avg. block size 126235383 B)
Minimally replicated blocks: 4 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Wed May 28 06:20:21 EDT 2014 in 6 milliseconds
The filesystem under path '/user/training/weblog/' is HEALTHY
[training@elephant data]$ sudo -u hdfs hdfs fsck /user/training/weblog/ -files -blocks -locations
Connecting to namenode via http://elephant:50070
FSCK started by hdfs (auth:SIMPLE) from /10.132.69.56 for path /user/training/weblog/ at Wed May 28 06:24:00 EDT 2014
/user/training/weblog/ <dir>
/user/training/weblog/access_log 1009883064 bytes, 8 block(s): OK
0. BP-1751026810-10.132.69.56-1401254548411:blk_1073741920_1096 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.132.155.185:50010]
1. BP-1751026810-10.132.69.56-1401254548411:blk_1073741921_1097 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
2. BP-1751026810-10.132.69.56-1401254548411:blk_1073741922_1098 len=134217728 repl=3 [10.132.155.185:50010, 10.132.69.56:50010, 10.132.183.144:50010]
3. BP-1751026810-10.132.69.56-1401254548411:blk_1073741923_1100 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
4. BP-1751026810-10.132.69.56-1401254548411:blk_1073741924_1101 len=134217728 repl=3 [10.132.69.56:50010, 10.132.155.185:50010, 10.166.3.224:50010]
5. BP-1751026810-10.132.69.56-1401254548411:blk_1073741925_1102 len=134217728 repl=3 [10.132.69.56:50010, 10.166.3.224:50010, 10.132.183.144:50010]
6. BP-1751026810-10.132.69.56-1401254548411:blk_1073741926_1103 len=134217728 repl=3 [10.132.69.56:50010, 10.166.3.224:50010, 10.132.155.185:50010]
7. BP-1751026810-10.132.69.56-1401254548411:blk_1073741927_1104 len=70358968 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
Status: HEALTHY
Total size: 1009883064 B
Total dirs: 1
Total files: 1
Total symlinks: 0
Total blocks (validated): 8 (avg. block size 126235383 B)
Minimally replicated blocks: 8 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Wed May 28 06:24:00 EDT 2014 in 2 milliseconds
The filesystem under path '/user/training/weblog/' is HEALTHY
[training@elephant data]$ sudo -u hdfs hdfs fsck /user/training/weblog/ -files -blocks -locations
Connecting to namenode via http://elephant:50070
FSCK started by hdfs (auth:SIMPLE) from /10.132.69.56 for path /user/training/weblog/ at Wed May 28 06:27:17 EDT 2014
/user/training/weblog/ <dir>
/user/training/weblog/access_log 1514824596 bytes, 12 block(s): OK
0. BP-1751026810-10.132.69.56-1401254548411:blk_1073741920_1096 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.132.155.185:50010]
1. BP-1751026810-10.132.69.56-1401254548411:blk_1073741921_1097 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
2. BP-1751026810-10.132.69.56-1401254548411:blk_1073741922_1098 len=134217728 repl=3 [10.132.155.185:50010, 10.132.69.56:50010, 10.132.183.144:50010]
3. BP-1751026810-10.132.69.56-1401254548411:blk_1073741923_1100 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
4. BP-1751026810-10.132.69.56-1401254548411:blk_1073741924_1101 len=134217728 repl=3 [10.132.69.56:50010, 10.132.155.185:50010, 10.166.3.224:50010]
5. BP-1751026810-10.132.69.56-1401254548411:blk_1073741925_1102 len=134217728 repl=3 [10.132.69.56:50010, 10.166.3.224:50010, 10.132.183.144:50010]
6. BP-1751026810-10.132.69.56-1401254548411:blk_1073741926_1103 len=134217728 repl=3 [10.132.69.56:50010, 10.166.3.224:50010, 10.132.155.185:50010]
7. BP-1751026810-10.132.69.56-1401254548411:blk_1073741927_1105 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.166.3.224:50010]
8. BP-1751026810-10.132.69.56-1401254548411:blk_1073741928_1106 len=134217728 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.132.155.185:50010]
9. BP-1751026810-10.132.69.56-1401254548411:blk_1073741929_1107 len=134217728 repl=3 [10.166.3.224:50010, 10.132.69.56:50010, 10.132.155.185:50010]
10. BP-1751026810-10.132.69.56-1401254548411:blk_1073741930_1108 len=134217728 repl=3 [10.132.155.185:50010, 10.132.69.56:50010, 10.166.3.224:50010]
11. BP-1751026810-10.132.69.56-1401254548411:blk_1073741931_1109 len=38429588 repl=3 [10.132.69.56:50010, 10.132.183.144:50010, 10.132.155.185:50010]
Status: HEALTHY
Total size: 1514824596 B
Total dirs: 1
Total files: 1
Total symlinks: 0
Total blocks (validated): 12 (avg. block size 126235383 B)
Minimally replicated blocks: 12 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Wed May 28 06:27:17 EDT 2014 in 3 milliseconds
The filesystem under path '/user/training/weblog/' is HEALTHY
[training@elephant data]$ ^C
[training@elephant data]$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment