Ruby script that finds identical (md5) files in all subdirectories (recursive)

find-duplicate-files.rb

# This Ruby script (regardless of where it's located on the file system) recur-
# sively lists all duplicate files in the direcotry in which it's executed.

require 'digest/md5'

hash = {}

Dir.glob('**/*', File::FNM_DOTMATCH).each do |f|
  next if File.directory?(f)
  key = Digest::MD5.hexdigest(IO.read(f)).to_sym
  if hash.has_key?(key) then hash[key].push(f) else hash[key] = [f] end
end

hash.each_value do |a|
  next if a.length == 1
  puts '=== Identical Files ==='
  a.each { |f| puts "\t" + f }
end

pretty cool idea to use the md5 hash for checks.

https://gist.github.com/costa/e12e853c77f9fee2714c FYI. Thanks!

https://gist.github.com/mailethiesen/146e4d9e9f54dc0bfbc02d104960dbf2 ME TOO! Thank you!

As a one-liner, using a few new things:

files = Pathname(".").glob("**/*").reject(&:directory?); d = files.map.with_index {|f, i| puts "%i / %i" % [i, files.size] if i % 1000 == 12; [f, Digest::MD5.file(f)] }.group_by(&:last).select {|a,v| v.size > 1 }