Skip to content

Instantly share code, notes, and snippets.

@thuvh
Forked from thattommyhall/clean_hdfs_tmp.rb
Last active August 29, 2015 14:17
Show Gist options
  • Save thuvh/509aaa8d14a506e2cf3f to your computer and use it in GitHub Desktop.
Save thuvh/509aaa8d14a506e2cf3f to your computer and use it in GitHub Desktop.
#!/usr/bin/env ruby
require "date"
five_days_ago = Date.parse(Time.now.to_s) - 5
IO.popen("hadoop fs -lsr /tmp").each_line do |line|
permissions,replication,user,group,size,mod_date,mod_time,path = *line.split(/\s+/)
if (mod_date)
if Date.parse(mod_date.to_s) < five_days_ago
puts line
if permissions.split('')[0] == 'd'
puts "deleting #{path}"
`hadoop fs -rmr -skipTrash #{path}`
dirname = path
next
end
next if path.start_with? dirname
`hadoop fs -rm -skipTrash #{path}`
end
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment