Skip to content

Instantly share code, notes, and snippets.

@cbaenziger
Last active Aug 30, 2018
Embed
What would you like to do?
JRuby File Deletion Script
include Java
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import java.io.FileNotFoundException.hadoop.fs.Path
import java.util.NoSuchElementException
# hdfs file system handle
fs = FileSystem.newInstance(Configuration.new)
list_of_files = '/tmp/file_list.txt'
delete_files = false
totals = {}
totals.default = 0
files = {'seen' => 0, 'found' => 0}
File.open(list_of_files) do |f|
f.each do |p|
path = Path.new(p.strip)
puts p
files['seen'] += 1
begin
if fs.isDirectory(path)
status = fs.listFiles(path, true)
while ( l = status.has_next()) do
l = status.next()
begin
totals[l.getOwner().to_s] = totals[status.getOwner()] + (l.getLen() * l.getReplication())
rescue java.io.FileNotFoundException => e
puts e.message
end
end
else
status = fs.listStatus(path)[0]
totals[status.getOwner().to_s] = totals[status.getOwner()] + (status.getLen() * status.getReplication())
end
rescue java.io.FileNotFoundException => e
puts e.message
next
end
files['found'] += 1
if delete_files == true
fs.delete(path, true)
end
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment